Evaluation of the project “Set-up of a demand-oriented and regionally harmonized quality infrastructure in the Andean Region”
CEval was mandated by Physikalisch Technische Bundesanstalt (PTB) to evaluate the project Set-up of a demand-oriented and regionally harmonized quality infrastructure in the Andean Region.
The project involved the countries of Bolivia, Ecuador, Colombia and Peru and aimed at improving the services offered by local institutions of quality infrastructure (QI). The evaluation addressed the criteria developed by DAC (relevance, effectiveness, impact, efficiency, sustainability) and the components of the management model Capacity Works (strategy, cooperation, steering structure, processes, learning and innovation). Specific evaluation questions were:
– How successful was the approach to develop regional QI-services?
– To what extent do the partners consider the approach as appropriate?
– Which success factors and lessons learnt can be identified?
– How successful was the approach to develop sector-specific supply strategies in each country?
– Which success factors and lessons learnt can be identified?
First, the evaluation consulted project documents only to assess the possibility to conduct evaluations based solely on documents. After a first assessment of the results, these were complemented by interviews with project staff of the PTB and its partner institutions. >From this, recommendations for future projects and evaluations based solely on documents were derived.
Meta-Evaluation: Project Evaluations of international development co-operation of the Friedrich-Ebert-Foundation (FES)
The International Development Cooperation Department (IEZ) of the Friedrich-Ebert-Stiftung (FES) commissions around ten external evaluations per year in order to continuously improve its own work, to initiate learning processes as an organization based on experience gained and to fulfil its accountability to the donor.
As part of a meta-evaluation, the Center for Evaluation (CEval) was commissioned to analyze 18 evaluation reports that had been prepared between 2012 and 2014 during a phase in which IEZ’s quality management had undergone significant development. The aim of the meta-evaluation was to identify systematic strengths and weaknesses of current evaluation practice, to derive findings for improving the quality of future evaluations and thus to increase their usefulness for project managers and the responsible units.
While the quality of the evaluation reports was analyzed in a desk study, in-depth expert interviews and focus group discussions were also held within the FES to assess the usefulness of the evaluations.
Improving the monitoring and reporting system to implement the action plan Inclusion of Persons with disabilities in the German development co-operation
In February 2013, the Federal Ministry for Economic Cooperation and Development (BMZ) published an action plan for the inclusion of persons with disabilities (PWDs). In doing so, it follows its duties emanating from Article 32 of the UN Convention on the rights of PWDs. The action plan provides information on strategic goals and actions to design development cooperation of the Federal Republic of Germany towards a better inclusion of PWDs.
To asses the Progress and to steer the implementation of planned an appropriate Monitoring and reporting system was necessary. Hence, the Center for Evaluation (CEval) had been contracted to Support the sector project “Inclusion of PWDs” of the German Agency for International Cooperation (GIZ) and the department 300 of the BMZ with its methodological Expertise to develop an Analysis grid to evaluate the implementation of the action plan.
Support for the compilation of possible methodological approaches to evaluating medium-term and long-term impacts of (policy) consulting in the specific context of the ECLAC project
In July 2014, the Deutsche Gesellschaft für Internationale Zusammenarbeit (GIZ) started the consultancy project Structural change for sustainable and inclusive development in Latin America and the Caribbean together with the United Nations Economic Commission for Latin America and the Caribbean (ECLAC). Currently, the project utilizes a monitoring system which allows tracking activities and consultancies carried out by ECLAC (inputs) and their utilization (outputs). Long-term impacts are not considered yet in this approach.
The Hertie-School of Governance (HSoG) was commissioned to elaborate possible methodological approaches to evaluating medium-term and long-term impacts of (policy) consulting in the specific context of the ECLAC project. Besides experience in policy consulting, this process also requires expertise in evaluation research and practice; therefore, the HSoG approached the Center for Evaluation (CEVAL) for collaboration.The consultancy comprises three work packages:
- A general review of theoretical approaches to the evaluation of policy consulting (considering qualitative methods of empirical social research);
- A review of practical experiences from policy consulting, if possible in similar contexts;
- An outline of a methodological approach to measuring medium-term and long-term impacts based on the results of the first two work packages and adapted to the specific context of the collaboration with ECLAC. CEval primarily contributed to the third work package and developed a methodological approach which combines a overarching impact and program specific outcome M&E. The approach proposed comprises a mixture of methods such as document analysis, standardized (online) surveys and guided interviews with a panel of experts.
Impact evaluation of the young leaders program “Agentes de Cambio” in Central America and the Dominican Republic
The Friedrich Ebert Foundation (FES) contracted the Center for Evaluation to evaluate their young leaders program “Agentes de Cambio”, which forms an integral part of the Foundations work and is active in countries such as Costa Rica, Dominican Republic, Guatemala, Honduras, El Salvador, Nicaragua and Panama. The evaluation aims at understanding the impact of this program as a whole and at identifying country-specific characteristics. Based on these insights, it will provide recommendations for further improvements of the curriculum, adaptations of the projects objectives and strategies and for improvement of steering mechanisms. Furthermore, the FES will be provided with a practical data collection tool for program monitoring and evaluation in the participating countries.
In this evaluation, the CEval used a quasi-experimental approach and combines semi-standardized online surveys with focus group discussions and expert interviews.
Final Evaluation Evaluation of SCAMPIS Scaling up micro-irrigation systems in India, Madagascar and Guatemala
Since 1977, IFAD has worked to raise productivity and income of rural poor populations to overcome poverty and improve the quality of their lives. In its long history, the Fund has developed and implemented various programs and projects all over the world. Scaling up micro-irrigation systems in India, Madagascar and Guatemala (SCAMPIS) is one of the IFAD’s projects, financed by COOPERNIC with a budget of 3 million Euro. In the period of 2008-2012, it has aimed at improving livelihoods and food security of 30.000 smallholder households in the three pilot countries of India, Madagascar and Guatemala by providing them with micro irrigation systems (MIS) and liquid organic fertilizer systems (LOF).
Before the program phased out in December 2012, an external evaluation of SCAMPIS was conducted in order to assess the achievement of program objectives and outcomes. In particular, according to the Terms of Reference, the evaluation had pursued three objectives:
- The review assessed the achievement of the programme objectives and outcomes with regard to its components and corresponding target groups.
- Lessons learnt and recommendations were provided for IFAD with regard to improvement of design and implementation of future programs. Replicability and scaling up were core concepts in this context.
- Special attention was put on the sustainability especially with regard to any future activities to be implemented by the different stakeholders along the water value chain of MIS/LOF.
The CEval evaluation team visited each project country for 12 days to conduct interviews with main stakeholders in the implementing organizations, key persons along the MIS value chain and the primary beneficiaries. In addition, a quantitative survey had been conducted in each of the three project countries to learn more about the impact MIS has on food security in different country contexts. Finally, IFADs perspective was taken into account by using data from interviews with most important representatives for the SCAMPIS project.
East4South – Development Awareness from the Roots
Subject of the evaluation was the EU-funded project of the Deutsche Welle Academy (DWA) East4South – Development Awareness from the Roots”. The projects overall goal was to raise awareness about development issues among EU-12 citizens and positively changing their attitude towards EU development policy. The project aimed specifically at enabling the EU12 journalists to prepare high quality media products covering development issues and to publish them in their home countries media by providing training in intercultural journalism and on-site research and a media production stay in Africa for young journalists from EU-12 countries. In the course of the project a total of 60 young journalists from Eastern Europe and Africa were fostered and supported when creating their own network.
Goal of the summative ex-post evaluation was to verify and analyze the outcome of the project activities and the quality of the project management. The CEval GmbH followed a mixed-method approach combining in-depth stakeholder interviews and secondary and document analysis of module evaluation surveys and participants short reports with a standardized online survey of all participants.Talking about migrants lives in Russia
Subject of Evaluation was the EU-funded project Talking about migrants lives in Russia of the DW Akademie and the Foundation for Independent Radio Broadcasting. The project aimed to increase awareness about the situation of migrants within Russian society by strengthening cooperation between the Russian media and civil society organizations (CSOs) who are working on migrant issues. In order to achieve this goal, training activities for radio journalists and CSO activists were provided. The workshops aimed to introduce the CSO activists to agenda-setting mechanism of the media and to equip the journalists with storytelling skills and in-the-field reporting skills in order to do genuine social reporting on migration issues. Following the training activities the workshop participants took part in panel discussions where they presented their media productions and exchanged experiences on the reporting on migrant issues. A major medium-term outcome of those activities was the establishment of a sustainable network between CSO activists and radio journalists.
Goal of the summative ex-post evaluation was to analyze and assess the short- and medium-term outcomes of the training workshops. For this purpose the evaluation additionally analyzed to what extent the projects web platform, the panel discussions and organizational aspects in general influenced the expected outcomes. For data acquisition, CEval GmbH followed a mixed-method approach combining in-depth stakeholder interviews, document as well as secondary analysis of module evaluation surveys. Additionally a standardized online survey in Russian of all workshop participants was carried out.Strengthening evaluation capacities within government and civil society in Uganda (ECD Uganda): Development of a blended-learning Master of Evaluation (2012-2015)
Evaluation as a scientific means to describe and assess interventions is gaining relevance worldwide. The needs for qualified evaluation professionals are increasing in Uganda, too. However, at the moment these needs cannot be satisfied due to lack of training- and evaluation programs in this region. It is for this reason that the BMZ together with CEval and GIZ initiated a program for developing evaluation capacities in Uganda.
The most important component of this program is the development and implementation of a master course in evaluation in the blended-learning format at Uganda Technology and Management University (UTAMU) in Kampala, Uganda. Blended-learning is an innovative concept which is gaining popularity worldwide and is now the default means of teaching at UTAMU. Blended-learning unites traditional face-to-face classes with modern forms of e-learning in a didactical coherent way. An advantage of this concept is, that it is possible not only for Ugandan students to attend classes but also for all students of English-speaking African countries.
The Master of Evaluation of Saarland University in cooperation with HTW Saarland serves as a template for the Ugandan Master, which will be introduced in fall 2015. The contents of the master program have been developed in cooperation with CEval GmbH, the teachers of the German master as well as Ugandan contact persons of Makarere University and have been adapted to the Ugandan context. The Ugandan Master of Monitoring and Evaluation will comprise 15 modules in total, seven of which are in compliance with the basic teaching program of evaluation at Saarland universities. This is to ensure that all topics relevant to acquire profound knowledge in the field of evaluation are covered. UTAMU has complemented this program with further modules regarding project management, monitoring as well as business start-up consulting. The modules are roughly comprised of web-based trainings, e-tutoring and face-to-face seminars. The teaching concepts have also been developed in cooperation with all stakeholders. The GIZ is responsible for the technical implementation of the e-learning concepts.
Two modules of the master of evaluation have been tested in an experimental phase together with UTAMU. During this experimental phase e-tutoring as well as the face-to-face seminars have been supervised in the tandem-principle by teachers of UTAMU and Saarland University/HTW. The hosting of the online study course has been undertaken by the GIZ platform GC 21. In a next step, the seven modules have been integrated into the already existing UTAMU master and the new course of study has been submitted to the responsible committees. The course of study is now aggregated and will be introduced experimentally. A decision about whether the course of study will be continued and which modifications would be necessary for this procedure, will be made in December 2016.
Strengthening of evaluation capacity in Central America (FOCEVAL)
The project “Strengthening of evaluation capacity in Central America (FOCEVAL)” of the Deutsche Gesellschaft für Internationale Zusammenarbeit (GIZ) GmbH aimed to strengthen the role of evaluation in the policy-making process and the control of policies in Costa Rica and selected countries in Central America. The measure was designed as a regional project, whose political sponsorship lies with the Costa Rican Ministry of National Planning and Economic Policy (MIDEPLAN).
Within the framework of the ECD-project, the task of the CEval GmbH was the implementation of a number of training courses on monitoring and evaluation (M&E) at different aspiration levels for the partner structures. The contents of the already tested and repeatedly conducted training courses reflect the different needs among the target groups. In addition, the CEval GmbH offered the regional project a continuous scientific support, which ensures a permanent connection to the current international discussion on evaluation, and maintains the link to scientific organizations and networks. It furthermore comprised a range of advisory services, particularly in the strategy and concept development, which was based on the latest scientific findings of monitoring and evaluation research.