Evaluation of the Funding Activities of the Foundation Environment and Development North Rhine-Westphalia
The Foundation Environment and Development North Rhine-Westphalia was founded 2001 and promotes mainly volunteer environmental and development work of civil society groups. The objective of the foundation is to support the environmental and development work in North Rhine-Westphalia (NRW) in order to establish this work in the consciousness and in the practical involvement of the population permanently and at the state of the art.
For this purpose about 1,100 projects of non-profit organisations in NRW were supported with a volume of EUR 47.5 million so far. In addition, the Foundation carries out its own projects, to also promote the implementation of Agenda 21 of the United Nations Conference on Environment and Development in Rio de Janeiro in in 1994.
Within the evaluation, the results and effects of the funding activities of the Foundation were recorded and assessed. Furthermore, the evaluation was intended to find out information on potential inhibitory and / or promoting factors in the funding activities of the Foundation with respect to the foundation goals. Based on these findings recommendations on priorities of the Foundation with regard to content, actors, target groups and the methods and formats will be derived.
CEval applied a mixed method approach to collect relevant data. Therefore, relevant documents and accessible data of the foundation and of funded projects will be analysed. This will be combined with data based on qualitative guideline-based interviews with several stakeholders and standardised online surveys of current and completed projects.
Assessing Effectiveness and Impact of the Extractive Industries Transparency Initiative (EITI)
The Extractive Industries Transparency Initiative (EITI) is a global standard to promote open and accountable management of natural resources. It seeks to strengthen government and company systems, inform public debate, and enhance trust. In each implementing country it is supported by a coalition of governments, companies and civil society working together.
The GIZ has mandated an evaluation of the initiative based on the following objectives:
- Evaluate the outcomes, effectiveness and impact (positive and negative, intended and unintended) of EITI against the backdrop of the EITI MDTF Results Framework, and to shed light on the links between transparency, accountability and, possibly, wider development outcomes.
- Assess EITIs theory of change and the EITI MDTF Results Framework in order to develop an improved generic EITI Results Model and assist selected MSGs to develop specific national EITI Results Models and to derive impact-oriented indicators from it.
These objectives and the complexity of EITIs context required a tailor-made design. The methodology combined
- macro-quantitative statistical analyses
- in-depth case studies
- participatory approaches to operationalization, data collection and analysis of findings.
Under the coordination of LBN Strategies and in collaboration with further consultants, the Center for Evaluation took the lead in the study component Approaches and Methodology.
Case Study for Thuringia in the course of the Evaluation of the European Dynamics for DEAR Efficiency Project
CEval was mandated by Planète Publique (Paris) with the accomplishment of a case study in the course of the evaluation of the EU-project European Dynamics for DEAR Efficiency. The project aimed on supporting developmental education work in four European regions (Thuringia/ Germany, Picardie/ France, Marches/ Italy und Luxembourg/ Belgium) as well as encouraging the exchange and mutual learning of the participating actors. In Thuringia, different developmental education measures and networking activities on a regional and European level were fostered. For the creation of the case study CEval was carrying out guideline based expert interviews with different project stakeholder.
Meta-Evaluation World Vision Germany
For the second time World Vision Germany (WVG) has engaged CEval to conduct a meta-evaluation of 29 evaluation reports of its long-term Area Development Programmes. The aim of the evaluation was to analyze the reports with regard to their quality and validity. Beyond this assessment, CEval provided recommendations to enhance the design and implementation of future evaluation. Based on the OECD/DAC evaluation criteria relevance, effectiveness, efficiency, impact and sustainability, CEval implemented a two-tier analysis method. According to WVGs quality criteria (i.e. voice and inclusion, transparency, appropriateness of evaluation methods, methodology, triangulation, and contribution) we first assessed to which extent the evaluation reports complied with various sub-criteria within one criterion. In a second step, we consolidated findings and deviated one aggregated rating per criterion. CEval additionally examined the criteria satisfaction of information needs and organization of findings/conceptualization of change.
This year, additionally an online survey investigating the utility, usage and usefulness of evaluation findings within WV has been conducted. The semi-standardized questionnaire contained questions concerning the quality of evaluation reports against several criteria, similar to the above mentioned ones to compare the perception of quality within WV with the results from the document analysis. Furthermore, qualitative data has been collected to get insights into the usage, utility and usefulness of evaluation results.
Moreover, CEval evaluated the transparency of WVGs published impact report. It critically reviewed the report and rated it against the following criteria: comprehensibility, presentation of the relevant information for the target group, methodological approach, validity of the used data and sources, objectivity, and access.
Meta-evaluation of project evaluations 2014-2015
Misereor supports a large number of development projects that are carried out by partner organizations. The evaluation of these projects, commissioned by the partner organizations or by Misereor, makes an important contribution to information and learning. Together with Evaluation.consulting, CEval was commissioned to carry out a meta-evaluation of the evaluations from 2014-2015. The aim was to collect meaningful information on the quality of the evaluations commissioned by partners and thus bring about possible improvements in the area of quality assurance.
The meta-evaluation was essentially based on the DAC standards and principles. Methodologically, a representative number of studies were subjected to a baseline check in the first stage in order to be able to draw quality conclusions from the breadth of the studies. The criteria of completeness and appropriateness of the documentation, independence, credibility and usefulness were assessed on the basis of various sub-criteria and then aggregated.
In a second step, some of the studies that fulfill a special learning function were subjected to an intensive analysis of these and other criteria. Mainly the evaluation reports were analyzed, but also related documents such as terms of reference and general Misereor documents on the topic. In addition, individual interviews were also conducted to answer specific research questions.
Accompanying and consulting of the impact assessment/baseline update of HEKS Bangladesh country program
HEKS/EPER’s Bangladesh 2013-1016 country programme aimed to promote the social inclusion of marginalized and disenfranchised minorities in north-western Bangladesh. The various interventions primarily targeted urban and rural Dalit and lowland Adibashi communities, who suffer from complex forms of discrimination. The exclusion of both minorities is a key cause of conflict between minorities and mainstream society. HEKS wanted to promote the social inclusion of the two groups with a human rights approach and a value chain approach.
HEKS wanted to measure the impact of the last project phase and at the same time collect valid baseline data for the next program phase. The CEval supported HEKS in this. The impact of the program was to be recorded with the help of household surveys and the qualitative method of contribution analysis. CEval mainly supported HEKS in the preparation of the quantitative data collection and the quantitative data analysis. CEval prepared the sampling strategy, helped with the development of the survey instruments and with the statistical analysis of the data. In the latter, it was particularly important to combine the previously collected data with the newly collected data in such a way that valid conclusions could be drawn from the statistical analysis.
Monitoring & Evaluation of slum rehabilitation project in Brazil
Jointly with a local partner, Knorr-Bremse Global Care e.V. (KBGC) supports a training and rehabilitation project in a community in Grajaú, a slum district in Sao Paolo, Brasil. The project aims at improving the living conditions of the community by qualifying young people for self-sustained income-generation and rehabilitating the area by building wooden houses. Main activities include training of youth in civil construction and set up of sustainable housing by residents of the slum and the trained youth.
KBGC commissioned CEval to establish a Monitoring & Evaluation framework to gather feedback on the training activities and trace quality of life outcomes over time to assess whether the programs objectives are being achieved. In order to identify the outcomes of the training on the qualification of the participants and their application of knowledge gained, a time-series design was applied, implying several phases of data collection with the help of standardized survey questionnaires. Outcomes of students were traced every six months over a period of three years, with baseline, monitoring and end line reports were drafted after each data collection to support organizational learning and improvement on project design.
Evaluation and Consultancy Service: The QUIZ-Compass under Critical Review
In 2006, the department of International Development Coorperation of Friedrich-Ebert-Foundation (FES) developed a document to improve Quality Management in international development cooperation, called the QUIZ-Compass. After several pilots and revisions the document became mandatory for all country projects in 2010. About the same time the Federal German Ministry for Economic Cooperation and Development (BMZ) began to work on an improved reporting and porposal System. Thus, in 2012 BMZ introduced new formats which became mandatory for all political foundations.
Given thes developments requirements by the QUIZ-Compass and by the BMZ regulations turned out to be partly overlapping and contradictory. Therefore FES decided to critically Review the QUIZ-Compass. The aim of the evaluation and consultancy Services was first to check whether it is still line with state of the art insights by the impact debate in development cooperation. Secondly, the practicability of the tool was tested. Finally, recommendations to enhance the QUIZ-Compass were derived.
Improving the monitoring and reporting system to implement the action plan Inclusion of Persons with disabilities in the German development co-operation
In February 2013, the Federal Ministry for Economic Cooperation and Development (BMZ) published an action plan for the inclusion of persons with disabilities (PWDs). In doing so, it follows its duties emanating from Article 32 of the UN Convention on the rights of PWDs. The action plan provides information on strategic goals and actions to design development cooperation of the Federal Republic of Germany towards a better inclusion of PWDs.
To asses the Progress and to steer the implementation of planned an appropriate Monitoring and reporting system was necessary. Hence, the Center for Evaluation (CEval) had been contracted to Support the sector project “Inclusion of PWDs” of the German Agency for International Cooperation (GIZ) and the department 300 of the BMZ with its methodological Expertise to develop an Analysis grid to evaluate the implementation of the action plan.
Support for the compilation of possible methodological approaches to evaluating medium-term and long-term impacts of (policy) consulting in the specific context of the ECLAC project
In July 2014, the Deutsche Gesellschaft für Internationale Zusammenarbeit (GIZ) started the consultancy project Structural change for sustainable and inclusive development in Latin America and the Caribbean together with the United Nations Economic Commission for Latin America and the Caribbean (ECLAC). Currently, the project utilizes a monitoring system which allows tracking activities and consultancies carried out by ECLAC (inputs) and their utilization (outputs). Long-term impacts are not considered yet in this approach.
The Hertie-School of Governance (HSoG) was commissioned to elaborate possible methodological approaches to evaluating medium-term and long-term impacts of (policy) consulting in the specific context of the ECLAC project. Besides experience in policy consulting, this process also requires expertise in evaluation research and practice; therefore, the HSoG approached the Center for Evaluation (CEVAL) for collaboration.The consultancy comprises three work packages:
- A general review of theoretical approaches to the evaluation of policy consulting (considering qualitative methods of empirical social research);
- A review of practical experiences from policy consulting, if possible in similar contexts;
- An outline of a methodological approach to measuring medium-term and long-term impacts based on the results of the first two work packages and adapted to the specific context of the collaboration with ECLAC. CEval primarily contributed to the third work package and developed a methodological approach which combines a overarching impact and program specific outcome M&E. The approach proposed comprises a mixture of methods such as document analysis, standardized (online) surveys and guided interviews with a panel of experts.