External Evaluation of the empowering people. Network
On behalf of Siemens Stiftung CEval GmbH implemented an external evaluation of the empowering people. Network. Founded in 2013, the network brings together inventors and social enterprises to develop solutions to address basic supply problems in fields such as health, food or education in developing countries. The empowering people. Award aims to identify simple and appropriate technological innovations with business potential and make them visible and accessible to everyone. As members of the empowering people. Network, currently 80 organizations that are implementing best-practice approaches receive long-term support in terms of organizational development, professionalization, and networking with partners.
In 2017, the empowering people. Network was evaluated for the first time since its foundation. The evaluation covered the entire lifetime of the network and its purpose was to account for progress and goals achieved so far and to gain lessons for strategic development. It therefore examined what impacts could be achieved by the respective modules and how these were assessed with regard to the overarching objectives. Leading questions of the evaluation were guided by the OECD-DAC evaluation criteria. It was characterized by a participative multi-methods approach and drew on contribution analysis that combined document/secondary analyses with qualitative (interviews, focus group discussions, case studies) as well as quantitative methods (partly standardized surveys). Lessons learnt and recommendations for the future development of the network were derived from the findings.
Impact Study: Assessing the Impact of Fairtrade on Poverty Reduction through Rural Development, Follow-up study 2017
TransFair e.V. aims at strengthening the objectives of Fairtrade and enabling producers in the Global South to make a living of the products they produce. Among other things, this should be achieved by establishing long-term and as direct as possible relations between producer organizations in the Global South and markets in the Global North and by complying with standards such as the legal minimum wage, wearing protective clothing, paid vacation and social insurance for workers working on plantations.
In order to assess the impact of Fairtrade on the socioeconomic environment of producers, TransFair e.V. and the Swiss Max Havelaar Foundation had commissioned CEval in 2011 to carry out the impact Evaluation “Assessing the Impact of Fairtrade on Poverty Reduction through Rural Development.” The study was the first of its kind to analyze the potential contribution of Fairtrade to improve the living conditions in rural areas. The study design covered the countries Ghana, India, Kenya and Peru and the product categories banana, cotton, flower, coffee, cocoa and tea.
2017, in its 25th anniversary year, TransFair e.V. wanted to follow up on the previous findings and commissioned CEval for the second time, enabling the assessment of longitudinal (comparison over time) and cross-sectional (FT and non-FT case study comparison) changes. A theory-based contribution analysis was applied, considering FTs Theory of Change and the indicators of the committee on sustainability assessment (COSA). Following a case study approach, the findings were mainly based on primary qualitative data collected via Focus Group Discussions, Key Informant Interviews and observations. A participatory quantitative seed assessment to assess perception changes over the last five years complemented the field research, which took place between October 2017 and January 2018. Findings were, eventually, structured along the economic, social and ecological dimension of the COSA indicators.
Formative Evaluation of the Out-of-School-Children Initiative (OOSCI)
The focus of this evaluation was the Out-of-School-Children Initiative (OOSCI) by UNICEF and UNESCO Institute for Statistics (UIS). The initiative was launched in 2010 to achieve a reduction in the number of children out of school in participating countries. Means to achieve this goal were to create comprehensive profiles of out-of-School children in each Country, identify the barriers that push children out of school and to propose changes in the policies and strategies of the partner country to address these barriers.
The evaluation aimed at testing the validity of the theory of change and its assumptions. It is used to strengthen the program logic and to provide a formative assessment of the progress towards the overall goal of achieving a reduction in out-of-school children. The evaluation also provides an in-depth insight into benefits for the different stakeholders and their involvement in the initiative. Furthermore, the findings of the evaluation are used to meet the accountability requirements by the donors of OOSCI.
Main source of information of this evaluation have been documents like OOSCI country studies, OOSCI regional studies, government plans and documents from other donors (e.g. the World Bank, UNESCO, foreign donors etc.). In a document review we assessed progress made through OOSCI for a sample of 42 countries and assessed the contents within the document using a 4-step coding. These codings were used for a descriptive analysis and additionally for a Qualitative Comparative Analysis (QCA). QCA enabled us to analyze thoroughly which set of conditions leads to which outcome. Further instrument of data collection has been an online survey to all country offices. Additional qualitative data has been collected in the US and during eight country visits. The evaluation has been conducted together with Proman and further Consultants. CEval took over the lead with regard to the evaluation methodology.
Meta-Evaluation of 34 evaluation reports of World Vision Deutschland
For the third time, World Vision Germany (WVG) commissioned CEval to conduct a meta-evaluation of evaluation reports of its supported development programmes. In total, 34 evaluation reports of the financial years of 2016 and 2017 were included, to analyze their quality and validity.
The meta-evaluation was based on an analysis matrix, jointly created by CEval and WVG, which is structured along the main criteria of voice and inclusion, transparency, methodology, triangulation, contribution, satisfaction of information needs, conceptualization of findings and the newly added criterion of sustainability. Firstly, criteria were assessed systematically along pre-defined sub-questions and corresponding scales. Secondly, results were then aggregated and visualized. Lastly, the results were compared with previous meta-evaluations and trends were discussed.
Midterm-Review of the FKPW program
In international comparison, German municipalities have comparably broad competencies due to their strong local self-administration. These competencies are also very valuable for development cooperation. Within the framework of municipal partnerships, German municipalities and their partner municipalities jointly carry out projects in municipalities of the Global South. In the process, technical knowledge is transferred from the German municipality to the partner municipality. The Service Agency Communities in One World (SKEW in German) offers various ways of supporting the developmental engagement of municipal partnerships.
One of these funding opportunities is the personnel funding instrument “Experts for Municipal Partnerships Worldwide” (FKPW in German). Within the framework of the FKPW program, municipal partnerships can apply for the placement of an expert in the partner municipality to support the creation of local structures and capacity development. The aim of the program is to enhance the value of the municipal partnership and the commitment of the German municipality through the placement of an expert.
CEval GmbH was commissioned by GIZ to conduct a mid-term review of the FKPW program. The aim of the review was to bring together the views of the different actors involved in the program and to give recommendations for the further development of the program. The main objective therefore was institutional learning on the part of the two implementing organizations, GIZ and SKEW. The methodology of the review was purely qualitative and based on the approach of the open capturing of perspectives. For the review, interviews and focus group discussions were conducted with representatives of the German municipalities, the partner municipalities, with the experts and the steering team of the programme, as well as with representatives of the responsible department of the German Federal Ministry for Economic Cooperation and Development (BMZ in German) and representatives of various departments within the organizations GIZ and SKEW.
Consultancy Services for the Development of a Project Monitoring and Evaluation Framework for ASEAN Cooperation Projects
CEval was commissioned by the Deutsche Gesellschaft für Internationale Zusammenarbeit (GIZ) to provide consultancy on the development of a Project Monitoring and Evaluation Framework (PMEF) for ASEAN Cooperation Projects, which is part of the ASEAN Secretariat Post 2015 Institutional Strengthening and Capacity Development project, an initiative under ASEAN-Germany cooperation implemented by GIZ. Prime task was to look into the ASEAN Secretariats corporate monitoring and evaluation processes and to provide expert support for refining the ASEAN Secretariats Project M&E framework. Special emphasis was put on providing support in harnessing and consolidating the monitoring and evaluation framework for ASEAN cooperation projects.
During the consultancy a number of stakeholder consultations as well as a three-day method training were conducted. The consultations and training served to improve the M&E capacities of key staff and to create a common discussion forum for different departments across the three pillars for exchanging views, and developing ownership and acceptance among the same.
The final report consolidates lessons learned, describes best practices within the sector, and develops concepts, possible mechanisms and recommendations on implementing a functional PMEF.
Meta-Evaluation World Vision Germany
For the second time World Vision Germany (WVG) has engaged CEval to conduct a meta-evaluation of 29 evaluation reports of its long-term Area Development Programmes. The aim of the evaluation was to analyze the reports with regard to their quality and validity. Beyond this assessment, CEval provided recommendations to enhance the design and implementation of future evaluation. Based on the OECD/DAC evaluation criteria relevance, effectiveness, efficiency, impact and sustainability, CEval implemented a two-tier analysis method. According to WVGs quality criteria (i.e. voice and inclusion, transparency, appropriateness of evaluation methods, methodology, triangulation, and contribution) we first assessed to which extent the evaluation reports complied with various sub-criteria within one criterion. In a second step, we consolidated findings and deviated one aggregated rating per criterion. CEval additionally examined the criteria satisfaction of information needs and organization of findings/conceptualization of change.
This year, additionally an online survey investigating the utility, usage and usefulness of evaluation findings within WV has been conducted. The semi-standardized questionnaire contained questions concerning the quality of evaluation reports against several criteria, similar to the above mentioned ones to compare the perception of quality within WV with the results from the document analysis. Furthermore, qualitative data has been collected to get insights into the usage, utility and usefulness of evaluation results.
Moreover, CEval evaluated the transparency of WVGs published impact report. It critically reviewed the report and rated it against the following criteria: comprehensibility, presentation of the relevant information for the target group, methodological approach, validity of the used data and sources, objectivity, and access.
Meta-evaluation of project evaluations 2014-2015
Misereor supports a large number of development projects that are carried out by partner organizations. The evaluation of these projects, commissioned by the partner organizations or by Misereor, makes an important contribution to information and learning. Together with Evaluation.consulting, CEval was commissioned to carry out a meta-evaluation of the evaluations from 2014-2015. The aim was to collect meaningful information on the quality of the evaluations commissioned by partners and thus bring about possible improvements in the area of quality assurance.
The meta-evaluation was essentially based on the DAC standards and principles. Methodologically, a representative number of studies were subjected to a baseline check in the first stage in order to be able to draw quality conclusions from the breadth of the studies. The criteria of completeness and appropriateness of the documentation, independence, credibility and usefulness were assessed on the basis of various sub-criteria and then aggregated.
In a second step, some of the studies that fulfill a special learning function were subjected to an intensive analysis of these and other criteria. Mainly the evaluation reports were analyzed, but also related documents such as terms of reference and general Misereor documents on the topic. In addition, individual interviews were also conducted to answer specific research questions.
Accompanying and consulting of the impact assessment/baseline update of HEKS Bangladesh country program
HEKS/EPER’s Bangladesh 2013-1016 country programme aimed to promote the social inclusion of marginalized and disenfranchised minorities in north-western Bangladesh. The various interventions primarily targeted urban and rural Dalit and lowland Adibashi communities, who suffer from complex forms of discrimination. The exclusion of both minorities is a key cause of conflict between minorities and mainstream society. HEKS wanted to promote the social inclusion of the two groups with a human rights approach and a value chain approach.
HEKS wanted to measure the impact of the last project phase and at the same time collect valid baseline data for the next program phase. The CEval supported HEKS in this. The impact of the program was to be recorded with the help of household surveys and the qualitative method of contribution analysis. CEval mainly supported HEKS in the preparation of the quantitative data collection and the quantitative data analysis. CEval prepared the sampling strategy, helped with the development of the survey instruments and with the statistical analysis of the data. In the latter, it was particularly important to combine the previously collected data with the newly collected data in such a way that valid conclusions could be drawn from the statistical analysis.
Monitoring & Evaluation of slum rehabilitation project in Brazil
Jointly with a local partner, Knorr-Bremse Global Care e.V. (KBGC) supports a training and rehabilitation project in a community in Grajaú, a slum district in Sao Paolo, Brasil. The project aims at improving the living conditions of the community by qualifying young people for self-sustained income-generation and rehabilitating the area by building wooden houses. Main activities include training of youth in civil construction and set up of sustainable housing by residents of the slum and the trained youth.
KBGC commissioned CEval to establish a Monitoring & Evaluation framework to gather feedback on the training activities and trace quality of life outcomes over time to assess whether the programs objectives are being achieved. In order to identify the outcomes of the training on the qualification of the participants and their application of knowledge gained, a time-series design was applied, implying several phases of data collection with the help of standardized survey questionnaires. Outcomes of students were traced every six months over a period of three years, with baseline, monitoring and end line reports were drafted after each data collection to support organizational learning and improvement on project design.