Energising Development (EnDev) stove assessment study in Ethiopia
The project Energising Development (EnDev), implemented by the GIZ, is a joint initiative of the governments of Australia, Germany, the Netherlands, Norway and the United Kingdom. Focus of this stove assessment study is the Energy Coordination Office (ECO), also known as EnDev Ethiopia. EnDev Ethiopia supports the development of a commercial stove sector and sustainable stove market not only by improving the business and technical skills of stove producers but also by strengthening the demand side of households.
The objective of this stove assessment study was therefore
- to analyze and assess the project intervention in three regions in Ethiopia by using the EnDev internal sustainability criteria and the HERA Sustainability Assessment Framework (HERA Poverty-Oriented Basic Energy Services);
- to generate recommendations for future steering of EnDevs project implementation.
Following a mixed methods approach, the study relied on standardised and qualitative methods of data collection in addition to the document analysis of relevant project and sector papers. CEval contributed to the study by providing its methodological expertise in data collection and analysis.
Meta-evaluation of 34 evaluation reports on the Area Development Programmes of World Vision Germany
World Vision Germany (WVG) had engaged CEval to conduct a meta-evaluation of 34 evaluation reports of its Area Development Programmes. The aim of the evaluation was to analyze the reports with regard to their quality and validity. Beyond this assessment, CEval provided recommendations to enhance the design and implementation of future evaluation.
Based on the evaluation criteria relevance, effectiveness, efficiency, impact and sustainability, CEval implemented a two-tier analysis method. According to WVGs quality criteria (i.e. voice and inclusion, transparency, appropriateness of evaluation methods, methodology, triangulation, and contribution) it had been first assessed to which extent the evaluation reports comply with various sub-criteria within one criterion. In a second step, findings had been consolidated and deviated one aggregated rating per criterion.
Moreover, CEval supported WVG in the production of their impact reports by critical commenting and methodological counselling.
Evaluation of the German Egyptian Research Fund
On behalf of the Federal Ministry for Education and Research the Project Management Agency of the German Aerospace Centre (DLR) implements the programme activities of the German-Egyptian Research Fund (GERF). The programme activities that had been evaluated comprise two rounds of funding (GERF I: 2009-2011, GERF II: 2011-2013/14) supporting German-Egyptian research projects in the fields of Biotechnology, Energy Efficiency, Renewable Energies, Water Desalination, Health, and Natural Resource Management.
CEval was contracted to conduct both, an ex-post evaluation of GERF I and a mid-term evaluation of GERF II. Accordingly, the objectives and downstream effects achieved by GERF I and their sustainability had to be evaluated, and the implementation process of the ongoing GERF II program had to be examined with regard to its design and efficiency, and recommendations for the continuation of GERF II and the design of future funding cycles had to be developed on the basis of the findings.
Final Evaluation Evaluation of SCAMPIS Scaling up micro-irrigation systems in India, Madagascar and Guatemala
Since 1977, IFAD has worked to raise productivity and income of rural poor populations to overcome poverty and improve the quality of their lives. In its long history, the Fund has developed and implemented various programs and projects all over the world. Scaling up micro-irrigation systems in India, Madagascar and Guatemala (SCAMPIS) is one of the IFAD’s projects, financed by COOPERNIC with a budget of 3 million Euro. In the period of 2008-2012, it has aimed at improving livelihoods and food security of 30.000 smallholder households in the three pilot countries of India, Madagascar and Guatemala by providing them with micro irrigation systems (MIS) and liquid organic fertilizer systems (LOF).
Before the program phased out in December 2012, an external evaluation of SCAMPIS was conducted in order to assess the achievement of program objectives and outcomes. In particular, according to the Terms of Reference, the evaluation had pursued three objectives:
- The review assessed the achievement of the programme objectives and outcomes with regard to its components and corresponding target groups.
- Lessons learnt and recommendations were provided for IFAD with regard to improvement of design and implementation of future programs. Replicability and scaling up were core concepts in this context.
- Special attention was put on the sustainability especially with regard to any future activities to be implemented by the different stakeholders along the water value chain of MIS/LOF.
The CEval evaluation team visited each project country for 12 days to conduct interviews with main stakeholders in the implementing organizations, key persons along the MIS value chain and the primary beneficiaries. In addition, a quantitative survey had been conducted in each of the three project countries to learn more about the impact MIS has on food security in different country contexts. Finally, IFADs perspective was taken into account by using data from interviews with most important representatives for the SCAMPIS project.
East4South – Development Awareness from the Roots
Subject of the evaluation was the EU-funded project of the Deutsche Welle Academy (DWA) East4South – Development Awareness from the Roots”. The projects overall goal was to raise awareness about development issues among EU-12 citizens and positively changing their attitude towards EU development policy. The project aimed specifically at enabling the EU12 journalists to prepare high quality media products covering development issues and to publish them in their home countries media by providing training in intercultural journalism and on-site research and a media production stay in Africa for young journalists from EU-12 countries. In the course of the project a total of 60 young journalists from Eastern Europe and Africa were fostered and supported when creating their own network.
Goal of the summative ex-post evaluation was to verify and analyze the outcome of the project activities and the quality of the project management. The CEval GmbH followed a mixed-method approach combining in-depth stakeholder interviews and secondary and document analysis of module evaluation surveys and participants short reports with a standardized online survey of all participants.Talking about migrants lives in Russia
Subject of Evaluation was the EU-funded project Talking about migrants lives in Russia of the DW Akademie and the Foundation for Independent Radio Broadcasting. The project aimed to increase awareness about the situation of migrants within Russian society by strengthening cooperation between the Russian media and civil society organizations (CSOs) who are working on migrant issues. In order to achieve this goal, training activities for radio journalists and CSO activists were provided. The workshops aimed to introduce the CSO activists to agenda-setting mechanism of the media and to equip the journalists with storytelling skills and in-the-field reporting skills in order to do genuine social reporting on migration issues. Following the training activities the workshop participants took part in panel discussions where they presented their media productions and exchanged experiences on the reporting on migrant issues. A major medium-term outcome of those activities was the establishment of a sustainable network between CSO activists and radio journalists.
Goal of the summative ex-post evaluation was to analyze and assess the short- and medium-term outcomes of the training workshops. For this purpose the evaluation additionally analyzed to what extent the projects web platform, the panel discussions and organizational aspects in general influenced the expected outcomes. For data acquisition, CEval GmbH followed a mixed-method approach combining in-depth stakeholder interviews, document as well as secondary analysis of module evaluation surveys. Additionally a standardized online survey in Russian of all workshop participants was carried out.Strengthening evaluation capacities within government and civil society in Uganda (ECD Uganda): Development of a blended-learning Master of Evaluation (2012-2015)
Evaluation as a scientific means to describe and assess interventions is gaining relevance worldwide. The needs for qualified evaluation professionals are increasing in Uganda, too. However, at the moment these needs cannot be satisfied due to lack of training- and evaluation programs in this region. It is for this reason that the BMZ together with CEval and GIZ initiated a program for developing evaluation capacities in Uganda.
The most important component of this program is the development and implementation of a master course in evaluation in the blended-learning format at Uganda Technology and Management University (UTAMU) in Kampala, Uganda. Blended-learning is an innovative concept which is gaining popularity worldwide and is now the default means of teaching at UTAMU. Blended-learning unites traditional face-to-face classes with modern forms of e-learning in a didactical coherent way. An advantage of this concept is, that it is possible not only for Ugandan students to attend classes but also for all students of English-speaking African countries.
The Master of Evaluation of Saarland University in cooperation with HTW Saarland serves as a template for the Ugandan Master, which will be introduced in fall 2015. The contents of the master program have been developed in cooperation with CEval GmbH, the teachers of the German master as well as Ugandan contact persons of Makarere University and have been adapted to the Ugandan context. The Ugandan Master of Monitoring and Evaluation will comprise 15 modules in total, seven of which are in compliance with the basic teaching program of evaluation at Saarland universities. This is to ensure that all topics relevant to acquire profound knowledge in the field of evaluation are covered. UTAMU has complemented this program with further modules regarding project management, monitoring as well as business start-up consulting. The modules are roughly comprised of web-based trainings, e-tutoring and face-to-face seminars. The teaching concepts have also been developed in cooperation with all stakeholders. The GIZ is responsible for the technical implementation of the e-learning concepts.
Two modules of the master of evaluation have been tested in an experimental phase together with UTAMU. During this experimental phase e-tutoring as well as the face-to-face seminars have been supervised in the tandem-principle by teachers of UTAMU and Saarland University/HTW. The hosting of the online study course has been undertaken by the GIZ platform GC 21. In a next step, the seven modules have been integrated into the already existing UTAMU master and the new course of study has been submitted to the responsible committees. The course of study is now aggregated and will be introduced experimentally. A decision about whether the course of study will be continued and which modifications would be necessary for this procedure, will be made in December 2016.
Strengthening of evaluation capacity in Central America (FOCEVAL)
The project “Strengthening of evaluation capacity in Central America (FOCEVAL)” of the Deutsche Gesellschaft für Internationale Zusammenarbeit (GIZ) GmbH aimed to strengthen the role of evaluation in the policy-making process and the control of policies in Costa Rica and selected countries in Central America. The measure was designed as a regional project, whose political sponsorship lies with the Costa Rican Ministry of National Planning and Economic Policy (MIDEPLAN).
Within the framework of the ECD-project, the task of the CEval GmbH was the implementation of a number of training courses on monitoring and evaluation (M&E) at different aspiration levels for the partner structures. The contents of the already tested and repeatedly conducted training courses reflect the different needs among the target groups. In addition, the CEval GmbH offered the regional project a continuous scientific support, which ensures a permanent connection to the current international discussion on evaluation, and maintains the link to scientific organizations and networks. It furthermore comprised a range of advisory services, particularly in the strategy and concept development, which was based on the latest scientific findings of monitoring and evaluation research.
Impact evaluation of the Hanns Seidel Foundation activities in the area of ‘civic education’ in Kenya
The Hanns Seidel Foundation (HSF) has been committed to development cooperation for more than 35 years (as of 2013) and has been working in Kenya since 1987. In 2013, the Foundation set out to conduct an impact evaluation of its activities in the area of civic education in Kenya for the first time. These activities first and foremost comprise civic education workshops.
This impact evaluation aimed at providing empirically and methodologically sound evidence of the HSFs civic education workshops in Kenya and making an estimation of the workshops expected sustainability.
To answer the evaluation questions an elaborate quasi-experimental design was chosen that was characterized by having several measurement times and by including two comparison groups. The different measurement times allow observing direct effects of the interventions as well as a (potential) change in these effects over time. The comparison group approach enabled us to control external effects on the target groups on the local and national levels. Data collection was implemented by a team of students of the University of Nairobi who have been trained for this specific task by the CEval.
Multi-Country Evaluation of Regional Knowledge and Leadership Areas: Area 4: Inclusion of all Out of School Children in Quality Learning in CEE/CIS
The purpose of the evaluation was (i) to assess the extent to which UNICEFs contributions to program interventions addressed major child rights violations in education, reduced equity gaps and led to impact results for children; and (ii) to draw from learning practices, innovations and models for future UNICEF country programming processes.
The objective is, therefore:
- to document and report on impact results in terms of changes in childrens life
- to assess and demonstrate how such results were made possible through system changes; and
- to document the contribution of UNICEF to these system changes.
In addition, the evaluation ensures that mechanisms for the assessment of impact results embrace issues of:
- contextual adequacy (regarding tailoring programming approaches to the qualitative differences between inequities);
- coordination (with other stakeholders);
- coherence (across policies and other supporting interventions); and
- protection (in terms of adequacy of the response of the respective education systems to protecting vulnerable groups).
Ultimately, the evaluation made a contribution towards improving future programming by adapting it to the specific groups of children out of school.
