Synthesis of three PTB technical cooperation instrument evaluations
CEval has been mandated by Physikalisch Technische Bundesanstalt (PTB) to synthesize the results of three PTB instrument evaluations. In 2014 PTB had commissioned separate evaluations of three of its technical cooperation instruments: CALIDENA, QuISP and NMI-Metrology User Relations, of which the first two evaluations have been implemented by CEval. Their findings had been aggregated, compared and contrasted in order to extract overarching success factors and lessons learnt and generate relevant recommendations, among others, with regard to the future incorporation of the three instruments into PTB technical cooperation.
To that end, CEval qualitatively analyzed the information contained in the evaluation reports and collected complementary information by conducting a number of qualitative expert interviews. The synthesis results had been presented in an evaluation report and a workshop had been held to present them to PTB management and staff.
Accompanying evaluation of the project ‘Media in Libya – Stability through Structure"
The „Media in Libya- Stability through Structure“ project had the main goal of supporting the introduction of modern media legislation in Libya and to strengthen the countrys media institutions. In addition, the project should contribute to strengthen independent and professional journalism in the provinces of Libya. The project had a total duration of 30 months and was funded by the European Commission within the SPRING program. The project was controlled and implemented by the Deutsche Welle Academy (Department of Media Development, Team Africa) in cooperation with the Institute for War and Peace Reporting (IWPR). The activities of the project were divided into four activity clusters, of which DW Academy three realised, while IWPR edited one. CEval evaluated in this project clusters 1 to 3, which were in responsibility of the DW Academy.
The evaluation of the project was divided into a baseline, an intermediate and a final evaluation at the end of the implementation phase. CEval followed a mixed-methods approach for data mining. An in-depth analysis of already existing documents was carried out as well as own surveys (standardized online surveys of the participants of the trainings and qualitative guideline interviews with external experts and officials DW Academy).
Conception, realization and analysis of an expert panel for the assessment of qualitative factors of the renewable energy sector in Chile
The Deutsche Gesellschaft für internationale Zusammenarbeit (GIZ), on behalf of the Federal Ministry for the Environment, Nature Conservation, Building and Nuclear Safety (BMUB), supports the Chilean Energy Ministry in the implementation of measures for the diversification of the countries energy supply. One component of this program is a project to promote the development of solar energy use. At the beginning of the project, the quantitative and qualitative factors influencing the effectivity and sustainability and therefore the achievement of the projects objectives needed to be identified. Particularly for the documentation of qualitative factors it was necessary to gather empirical data through an expert panel, i.e. structured interviews with energy experts.
In the context of this assignment, the CEval GmbH rendered the following services to GIZ:
– Concept development: Identification of experts for the panel in coordination with GIZ, development of structured questionnaires for each group of stakeholders, development of a concept for future periodic panel interviews, development of a methodological concept for the long-term continuation of the expert Panel
- Realization of the first data collection round
- Analysis and reporting of the first data collection round
- Development of a concept for the identification of quantitative impacts using the qualitative data, applying Stockmanns funnel approach
Evaluation of the project “Set-up of a demand-oriented and regionally harmonized quality infrastructure in the Andean Region”
CEval was mandated by Physikalisch Technische Bundesanstalt (PTB) to evaluate the project Set-up of a demand-oriented and regionally harmonized quality infrastructure in the Andean Region.
The project involved the countries of Bolivia, Ecuador, Colombia and Peru and aimed at improving the services offered by local institutions of quality infrastructure (QI). The evaluation addressed the criteria developed by DAC (relevance, effectiveness, impact, efficiency, sustainability) and the components of the management model Capacity Works (strategy, cooperation, steering structure, processes, learning and innovation). Specific evaluation questions were:
– How successful was the approach to develop regional QI-services?
– To what extent do the partners consider the approach as appropriate?
– Which success factors and lessons learnt can be identified?
– How successful was the approach to develop sector-specific supply strategies in each country?
– Which success factors and lessons learnt can be identified?
First, the evaluation consulted project documents only to assess the possibility to conduct evaluations based solely on documents. After a first assessment of the results, these were complemented by interviews with project staff of the PTB and its partner institutions. >From this, recommendations for future projects and evaluations based solely on documents were derived.
Meta-Evaluation: Project Evaluations of international development co-operation of the Friedrich-Ebert-Foundation (FES)
The International Development Cooperation Department (IEZ) of the Friedrich-Ebert-Stiftung (FES) commissions around ten external evaluations per year in order to continuously improve its own work, to initiate learning processes as an organization based on experience gained and to fulfil its accountability to the donor.
As part of a meta-evaluation, the Center for Evaluation (CEval) was commissioned to analyze 18 evaluation reports that had been prepared between 2012 and 2014 during a phase in which IEZ’s quality management had undergone significant development. The aim of the meta-evaluation was to identify systematic strengths and weaknesses of current evaluation practice, to derive findings for improving the quality of future evaluations and thus to increase their usefulness for project managers and the responsible units.
While the quality of the evaluation reports was analyzed in a desk study, in-depth expert interviews and focus group discussions were also held within the FES to assess the usefulness of the evaluations.
Improving the monitoring and reporting system to implement the action plan Inclusion of Persons with disabilities in the German development co-operation
In February 2013, the Federal Ministry for Economic Cooperation and Development (BMZ) published an action plan for the inclusion of persons with disabilities (PWDs). In doing so, it follows its duties emanating from Article 32 of the UN Convention on the rights of PWDs. The action plan provides information on strategic goals and actions to design development cooperation of the Federal Republic of Germany towards a better inclusion of PWDs.
To asses the Progress and to steer the implementation of planned an appropriate Monitoring and reporting system was necessary. Hence, the Center for Evaluation (CEval) had been contracted to Support the sector project “Inclusion of PWDs” of the German Agency for International Cooperation (GIZ) and the department 300 of the BMZ with its methodological Expertise to develop an Analysis grid to evaluate the implementation of the action plan.
Support for the compilation of possible methodological approaches to evaluating medium-term and long-term impacts of (policy) consulting in the specific context of the ECLAC project
In July 2014, the Deutsche Gesellschaft für Internationale Zusammenarbeit (GIZ) started the consultancy project Structural change for sustainable and inclusive development in Latin America and the Caribbean together with the United Nations Economic Commission for Latin America and the Caribbean (ECLAC). Currently, the project utilizes a monitoring system which allows tracking activities and consultancies carried out by ECLAC (inputs) and their utilization (outputs). Long-term impacts are not considered yet in this approach.
The Hertie-School of Governance (HSoG) was commissioned to elaborate possible methodological approaches to evaluating medium-term and long-term impacts of (policy) consulting in the specific context of the ECLAC project. Besides experience in policy consulting, this process also requires expertise in evaluation research and practice; therefore, the HSoG approached the Center for Evaluation (CEVAL) for collaboration.The consultancy comprises three work packages:
- A general review of theoretical approaches to the evaluation of policy consulting (considering qualitative methods of empirical social research);
- A review of practical experiences from policy consulting, if possible in similar contexts;
- An outline of a methodological approach to measuring medium-term and long-term impacts based on the results of the first two work packages and adapted to the specific context of the collaboration with ECLAC. CEval primarily contributed to the third work package and developed a methodological approach which combines a overarching impact and program specific outcome M&E. The approach proposed comprises a mixture of methods such as document analysis, standardized (online) surveys and guided interviews with a panel of experts.
Impact evaluation of the young leaders program “Agentes de Cambio” in Central America and the Dominican Republic
The Friedrich Ebert Foundation (FES) contracted the Center for Evaluation to evaluate their young leaders program “Agentes de Cambio”, which forms an integral part of the Foundations work and is active in countries such as Costa Rica, Dominican Republic, Guatemala, Honduras, El Salvador, Nicaragua and Panama. The evaluation aims at understanding the impact of this program as a whole and at identifying country-specific characteristics. Based on these insights, it will provide recommendations for further improvements of the curriculum, adaptations of the projects objectives and strategies and for improvement of steering mechanisms. Furthermore, the FES will be provided with a practical data collection tool for program monitoring and evaluation in the participating countries.
In this evaluation, the CEval used a quasi-experimental approach and combines semi-standardized online surveys with focus group discussions and expert interviews.
Development of an impact-orientated M&E-System in Germany for the Cooperation Platform of Latin America North (COPLAN)
Up to the present innovative environmental technologies are not sufficiently utilized in Mexico and Central America. One reason is the missing extension of modern environmental technologies in this region. That is the initial point of COPLAN (Cooperation Platform of Latin America North): The objective of COPLAN is the extension of (German) environmental technologies in Mexico and Central America to make a contribution to environment protection. In the last years the German environment economics has led the way worldwide and was responsible for many innovations in the area of application-oriented technologies. Three questions were the base of the survey, namely a) What is the export potential of the German industrial sector of environmental technologies? b) Correspond the German supply with the potential demand in Mexico and Central America? and c) Which export barriers exist and how they can be overcome?
Because of the intended project effects in Germany it was necessary to create another impact-orientated monitoring-system in Germany beside the local impact-orientated monitoring-system in Mexico. The central task was to create an impact-orientated Monitoring and Evaluation System for measuring the intended outcome of COPLAN in Germany. For this purpose the main task was to develop an extensive indicator raster and an investigation plan for implementing two panels measuring the changes of the two target groups, the German enterprises of environmental technologies on the one hand, and central multipliers of the German environment economics on the other hand. Following the development of the M&E system, the CEval was tasked with implementing the first wave of the panel design. This included quantitative as well as qualitative data collection methods (standardized survey with participating companies, a comparison group and multiplicators, analysis of documents and secondary data, guided expert interviews).
Energising Development (EnDev) stove assessment study in Ethiopia
The project Energising Development (EnDev), implemented by the GIZ, is a joint initiative of the governments of Australia, Germany, the Netherlands, Norway and the United Kingdom. Focus of this stove assessment study is the Energy Coordination Office (ECO), also known as EnDev Ethiopia. EnDev Ethiopia supports the development of a commercial stove sector and sustainable stove market not only by improving the business and technical skills of stove producers but also by strengthening the demand side of households.
The objective of this stove assessment study was therefore
- to analyze and assess the project intervention in three regions in Ethiopia by using the EnDev internal sustainability criteria and the HERA Sustainability Assessment Framework (HERA Poverty-Oriented Basic Energy Services);
- to generate recommendations for future steering of EnDevs project implementation.
Following a mixed methods approach, the study relied on standardised and qualitative methods of data collection in addition to the document analysis of relevant project and sector papers. CEval contributed to the study by providing its methodological expertise in data collection and analysis.
