Meta-evaluation of project evaluations 2014-2015
Misereor supports a large number of development projects that are carried out by partner organizations. The evaluation of these projects, commissioned by the partner organizations or by Misereor, makes an important contribution to information and learning. Together with Evaluation.consulting, CEval was commissioned to carry out a meta-evaluation of the evaluations from 2014-2015. The aim was to collect meaningful information on the quality of the evaluations commissioned by partners and thus bring about possible improvements in the area of quality assurance.
The meta-evaluation was essentially based on the DAC standards and principles. Methodologically, a representative number of studies were subjected to a baseline check in the first stage in order to be able to draw quality conclusions from the breadth of the studies. The criteria of completeness and appropriateness of the documentation, independence, credibility and usefulness were assessed on the basis of various sub-criteria and then aggregated.
In a second step, some of the studies that fulfill a special learning function were subjected to an intensive analysis of these and other criteria. Mainly the evaluation reports were analyzed, but also related documents such as terms of reference and general Misereor documents on the topic. In addition, individual interviews were also conducted to answer specific research questions.
Accompanying and consulting of the impact assessment/baseline update of HEKS Bangladesh country program
HEKS/EPER’s Bangladesh 2013-1016 country programme aimed to promote the social inclusion of marginalized and disenfranchised minorities in north-western Bangladesh. The various interventions primarily targeted urban and rural Dalit and lowland Adibashi communities, who suffer from complex forms of discrimination. The exclusion of both minorities is a key cause of conflict between minorities and mainstream society. HEKS wanted to promote the social inclusion of the two groups with a human rights approach and a value chain approach.
HEKS wanted to measure the impact of the last project phase and at the same time collect valid baseline data for the next program phase. The CEval supported HEKS in this. The impact of the program was to be recorded with the help of household surveys and the qualitative method of contribution analysis. CEval mainly supported HEKS in the preparation of the quantitative data collection and the quantitative data analysis. CEval prepared the sampling strategy, helped with the development of the survey instruments and with the statistical analysis of the data. In the latter, it was particularly important to combine the previously collected data with the newly collected data in such a way that valid conclusions could be drawn from the statistical analysis.
Monitoring & Evaluation of slum rehabilitation project in Brazil
Jointly with a local partner, Knorr-Bremse Global Care e.V. (KBGC) supports a training and rehabilitation project in a community in Grajaú, a slum district in Sao Paolo, Brasil. The project aims at improving the living conditions of the community by qualifying young people for self-sustained income-generation and rehabilitating the area by building wooden houses. Main activities include training of youth in civil construction and set up of sustainable housing by residents of the slum and the trained youth.
KBGC commissioned CEval to establish a Monitoring & Evaluation framework to gather feedback on the training activities and trace quality of life outcomes over time to assess whether the programs objectives are being achieved. In order to identify the outcomes of the training on the qualification of the participants and their application of knowledge gained, a time-series design was applied, implying several phases of data collection with the help of standardized survey questionnaires. Outcomes of students were traced every six months over a period of three years, with baseline, monitoring and end line reports were drafted after each data collection to support organizational learning and improvement on project design.
Evaluation and Consultancy Service: The QUIZ-Compass under Critical Review
In 2006, the department of International Development Coorperation of Friedrich-Ebert-Foundation (FES) developed a document to improve Quality Management in international development cooperation, called the QUIZ-Compass. After several pilots and revisions the document became mandatory for all country projects in 2010. About the same time the Federal German Ministry for Economic Cooperation and Development (BMZ) began to work on an improved reporting and porposal System. Thus, in 2012 BMZ introduced new formats which became mandatory for all political foundations.
Given thes developments requirements by the QUIZ-Compass and by the BMZ regulations turned out to be partly overlapping and contradictory. Therefore FES decided to critically Review the QUIZ-Compass. The aim of the evaluation and consultancy Services was first to check whether it is still line with state of the art insights by the impact debate in development cooperation. Secondly, the practicability of the tool was tested. Finally, recommendations to enhance the QUIZ-Compass were derived.
Compliation of methodological approaches and experiences for impact Evaluations of Training and capacity building measures in the public sector
In cooperation with the Costa Rican institute CICAP, CEval counselled the GIZ program CADESAN in the Andean region. The regional program supports the development of capacity building measures on decentralization and modernization in the public sector.
The consultancy aimed at providing GIZ and its partners with an analysis and compilation of possible methodological approaches to evaluate the impacts of capacity building and training measures. To that end, various approaches and evaluation designs, their applicability in the respective realm as well as international experiences and practices were analyzed. The results were discussed with GIZ and its partners during a training designed for that purpose.
Project Evaluation of GIZ’s Sector Project Inclusion of persons with Disabilities
As part of the project evaluation, the extent to which the Deutsche Gesellschaft für Internationale Zusammenarbeit (GIZ) sector programme Inclusion of Persons with Disabilities was able to achieve developmental impacts was examined. Based on the OECD-DAC criteria, expert interviews and focus group discussions were conducted and M&E data from the sector programme and other documents were systematically analysed. This contributed to accountability towards the German Federal Ministry for Economic Cooperation and Development (BMZ), the publication of a short report met the requirements of the International Aid Transparency Initiative (IATI) and laid the foundation for planning a follow-up measure.
Evaluation of the Initiative “Schulen – Partner der Zukunft” (PASCH)
The initiative Schools Partners of the future (PASCH), founded in 2008, is part of the central initiatives of the German foreign cultural and educational policy within the current legislature period. PASCH is a worldwide network currently including around 1800 partner schools, which are linked to Germany and especially teach German. The initiative concentrates diversified measures, which are supposed to contribute to establish lively and long-term relations to Germany, and encourage schools, teachers and students to exchange their knowledge and ideas as well as to work together. PASCH is coordinated by the German Federal Foreign Office and is implemented by the Central Office for Schools Abroad, the Goethe Institute, the Pedagogic Exchange Service, the Conference of Education Ministers and the German Academic Exchange Service.
Within the scope of the planned evaluation, PASCH will be analyzed regarding its relevance, efficiency, effectivity, impact and sustainability. It will also be analyzed to what extent the existing offers, processes and structures are suitable instruments for achieving the objectives. Recommendations for possible optimization of the measures, processes and structures and the creation of synergies will be derived on the basis of these results.
The evaluation is divided into two phases: During the first phase, document analyses as well as guided personal and telephone interviews with stakeholders in Germany and selected partner countries (Poland, Turkey and China) will be conducted. The focus of the second phase is then on quantitative online surveys with target groups in China, Turkey and Poland as well as on qualitative and quantitative data collection in another case study country.
Evaluation of the project ‘Media in Libya: Stability through reconciliation’
The object of this evaluation was the project ‘Media in Libya: Stability through reconciliation’ (MLSR) of the DW Akademie. MLSR was a follow-up of the precursor project Media in Libya: Stability through Structure. The overall goal of MLSR was to enable a process of public discourse and eventually reconciliation across a fragmented Libyan society prerequisites for development of democratic reform and constitutional process in the country. By promoting [the] right to access to information to the people of Libya and [the] freedom of expression in Libya (DW Akademie, 2017, p. 4). The approach of MLSR is fourfold (i.e., four activity clusters): (1) supporting Libyan institutions/authorities efforts in the field of media governance, (2) developing a Libyan Media Network, (3) training local freelance journalists from all regions of Libya with a special focus on ethnic minorities and gender, and (4) providing grants for Libyan media projects.
The evaluation of MLSR had been considered as a planned impact evaluation, which needs valid qualitative/quantitative data on the situation at the beginning of MLSR (i.e., baseline). The second stage of evaluation (i.e., interim evaluation) started in January 2019. Comparing the results of baseline and interim evaluation, the evaluators had been able to identify and assess the preliminary effects of MLSR. Moreover, the evaluators may give first recommendations on how to improve MLSR (e.g., identification of [un]successful project strategies, alternative actions). The third stage of evaluation started shortly after the end of MLSR (i.e., January 2021). The main goal of the final evaluation was to assess the effectiveness of MLSR (i.e., achievements in relevant targeted areas).
Methodologically, a robust mixed-method approach was pursued against the background of the very volatile and difficult framework conditions in Libya: Secondary analysis of existing documents/data, content analysis of short written and in-depth oral interviews and statistical analysis of quantitative data from a standardized questionnaire survey.
Evaluation of the German Egyptian Research Fund
On behalf of the Federal Ministry for Education and Research the Project Management Agency of the German Aerospace Centre (DLR) implements the programme activities of the German-Egyptian Research Fund (GERF). The programme activities that had been evaluated comprise two rounds of funding (GERF I: 2009-2011, GERF II: 2011-2013/14) supporting German-Egyptian research projects in the fields of Biotechnology, Energy Efficiency, Renewable Energies, Water Desalination, Health, and Natural Resource Management.
CEval was contracted to conduct both, an ex-post evaluation of GERF I and a mid-term evaluation of GERF II. Accordingly, the objectives and downstream effects achieved by GERF I and their sustainability had to be evaluated, and the implementation process of the ongoing GERF II program had to be examined with regard to its design and efficiency, and recommendations for the continuation of GERF II and the design of future funding cycles had to be developed on the basis of the findings.
External Evaluation of Four Different Projects in the Field of Awareness Raising and a Synthesis
As international Christian development organisation, the Christian Blind Mission (CBM) aims at improving the human rights situation and the inclusion of people with disabilities. Awareness raising is one important component to achieve this target successfully. To learn how awareness raising activities are implemented, how they contribute to CBMs overall aim and how project and country specifics produce different results, CBM Germany has engaged CEval to evaluate four projects and to compile findings and recommendations in a synthesis report.
With co-funding, either by the European Union or by the German Federal Ministry for Economic Cooperation and Development, CBM Germany implements two projects addressing the needs of people with hearing impairment in the Palestine Territories (Gaza and Gaza Strip), one project to improve eye care and rehabilitation in India, and one project to strengthen disability rights and inclusion in Swaziland. Given the diversity of these projects, issues of awareness raising can be studied from various perspectives. Thus, analysing the different underlying assumptions and needs which in turn lead to various approaches and expected outcomes, will contribute to a better understanding of the overall topic of awareness raising. While the projects in Gaza and India require a mid-term evaluation as they are still on-going, the projects in Gaza Strip and Swaziland are close to their end and will be finally evaluated