Support of the BfN in the Evaluation by the German Science and Humanities Council (WR) 2023/2024
As a federal departmental research institution, the Federal Agency for Nature Conservation (BfN) is evaluated at regular intervals by external experts. After the first external evaluation by the German Council of Science and Humanities (WR) in 2007 and the second evaluation in 2015, the third evaluation is now due in 2024. The aim of the project is to support the BfN in the conceptual preparation and follow-up of the upcoming evaluation process through external, scientific expertise.
The following core tasks will be fulfilled:
– Support in the collection, evaluation and preparation of data and information needed to answer the questions to the BfN and the preparation of the report;
– Support in the preparation and implementation of the inspection by the WR;
– Support in the analysis of the evaluation results and in the follow-up of the evaluation;
– Analysis of the effectiveness and quality of BfN’s science-based policy advice and knowledge communication, including the development of recommendations for its systematic further development. Development of proposals for the integration of these recommendations into the quality management of BfN’s scientific work.
Program evaluation of the federal program for biological diversity
The German National Strategy on biodiversity strives to improve the protection and sustainable use of biodiversity in Germany. The strategy was adopted by the federal government in 2007 and has been supplemented by the Nature Conservation Offensive 2020 since 2015.
Since 2011, the Federal Program for Biological Diversity has taken on a crucial role within the framework of the biodiversity strategy as Germany’s most financially powerful funding instrument for nature conservation projects. It promotes measures aimed at halting the decline in biodiversity in Germany and reversing it into a positive trend in the medium to long term.
To assess its success, the federal programme has established evaluations of individual projects. However, a systematic and aggregated analysis of the pool of projects funded within the framework of the programme was missing in its 10-year duration.
Therefore, CEval was working with adelphi and PAN to develop a structured, precisely fitting analysis methodology to systematically assess the quality of the programme.
For this purpose, documents from more than 60 completed or far advanced projects were systematically evaluated and complemented by expert interviews. In addition, further interviews were conducted at the level of programme management in order to identify potentials for the improvement of the operational and strategic programme steering. Overall, key aspects, successes, weaknesses and potentials at programme and project level were identified and systematically reflected to formulate recommendations for the improvement of future funding practices.
External Evaluation of Climate-friendly Communal Catering within the Project “MehrWertKonsum” of the North Rhine-Westphalia Consumer Advice Centre
The project MehrWertKonsum More appreciation of food and finite resources conducted by the consumer advice centre of North Rhine-Westphalia aims to reduce carbon dioxide emissions encouraging a responsible, sustainable and climate-friendly consumption. The primary objective of the project is to contribute to saving greenhouse gas emissions. Regarding communal catering, among other things that means to diminish leftovers and to offer more vegetarian dishes with seasonal and regional foods that are organically grown. Information and educational material will be passed to day care centres, schools, youth hostels and their food suppliers, waste will be measured, leftovers on plates and at the service counters will be analysed and suggestions for improvement will be developed. Furthermore, menus will be checked and enhanced regarding to their desirability as well as climate and environment protection. Schools, in addition, will have the possibility to get participative educational services and activities for their students. In youth hostels, there will be a special focus on the communication with the guests to encourage the acceptance of the measures and thus to contribute to waste prevention. The project is funded through the EFRE OP NRW, an operational programme of North Rhine-Westphalia concerning the promotion of investments in growth and employment supported by the European Regional Development Fund, and North Rhine-Westphalias Ministry of Environment. Within EFRE OP NRW projects greenhouse gas savings (t CO2 equivalents/year) have to be substantiated.
Moreover, the following objectives of the field of action climate-friendly communal catering had been evaluated:
– To reduce food and catering waste
– To establish climate-friendly and healthy catering offers
– To raise consciousness of and appreciation for food, climate-friendly nutrition and food waste
In the course of the external evaluation of MehrWertKonsum the impact of the project measures had been analyzed both on the outcome level (in particular altered behavior of the target group) as well as on the impact level (greenhouse gas savings). To do so, the evaluation team of CEval GmbH, schmidt evaluation and EBP applied a complex multi-method approach including participant observation, qualitative telephone interviews and quantitative online surveys. The results had been combined with the internal monitoring of the project MehrWertKonsum in a way that greenhouse gas savings could be calculated.
Evaluation of Individual Projects of the International Climate Initiative (IKI)
Since 2008, the International Climate Initiative (IKI) is funding initiatives regarding (i) climate protection, (ii) adaptation to the impacts of climate change, (iii) conserving natural carbon sinks and (iv) the protection of biological diversity in developing and newly industrialising countries on a project basis. Overall, the IKI now includes almost 600 individual projects and 2.8 billion euros in funding. The IKI focuses on climate protection which comprises almost half of the projects. There was a first evaluation conducted between 2010 and 2013 that included 115 projects. Within this evaluation, a full survey of the projects completed or almost completed by 2011 was realized. This project aimed to carry out further evaluations of those projects that were completed by the end of 2019. The individual project evaluations consisted of detailed analyses according to the OECD DAC criteria as well as other program-specific criteria. The focus of the results chain was on an analysis of inputs, outputs and outcomes. Impacts at higher levels were only analyzed on the basis of reliable information available at the time of the evaluations.
Results oriented monitoring of the technical cooperation measure COPLAN II and evaluation of the measure’s contribution towards a better alignment of development cooperation and foreign trade facilitation on a systemic level
Climate Change and the uninterrupted decline of fossil fuels affect different regions of the world in different ways. Many nations have thus formulated ambitious goals: Mexico for instance is determined to cover one third of its energy production by means of renewable energies by 2024. Despite such ambitions however environmental conservation has not been marked by major success to this date. This is partly due to a lack of dispersion of modern environmental technology. Against this background the GIZ Project Cooperation Platform Latin America North II (COPLAN II) scrutinizes central deficits that hinder international investment and cooperation such as a lack of professional expertise regarding modern environmental technology in small and medium enterprises (SMEs).
Within the scope of the present assignment CEval together with iSPO Institute supported COPLAN II in its monitoring and evaluation efforts. Starting point for any results oriented monitoring was typically an impact model with respective indicators and assumptions. Changes within the mechanisms of cooperation are however rarely or marginally observed under the monitoring component. This development of cooperation systems or the suitability of the relevant funding instruments are also examined as part of the evaluation component.
All in all, next to monitoring tasks and the compilation of results of individual measures the assignment at hand intended to also generate insights into COPLAN IIs contribution towards a better alignment of development cooperation and foreign trade facilitation as well as public-private sector cooperation on a systemic level.
Scientific support of an impact analysis of the project “egal war gestern – towards sustainability at vocational schools”
The Landesbund für Vogelschutz in Bayern (LBV) e.V. implemented the project Towards sustainability at vocational schools in the school term 2018/19 for the second time. It aimed at mobilizing the youth for matters of sustainability. More precisely, the project was geared towards students at Bavarian vocational schools who find themselves in the important period of professional orientation. The goal was to direct their attention to sustainability questions and to strengthen their awareness of those issues. To achieve this goal, the students were expected to independently implement projects in their respective schools in order to contribute to a more sustainable school life.
The project was subject to an impact analysis, which was jointly implemented by LBV and CEval GmbH within the context of an accompanying evaluation. The purpose was to determine whether the intended effects of the project were actually achieved, that is, whether the attitudes and behavior of the participating students actually changed in relation to sustainability matters and whether those matters are more readily integrated into the everyday life at Bavarian vocational schools. In terms of methodology the analysis was based on a standardized survey before and after the project with both the participating students and a comparison group of students. Furthermore, guided interviews with the teachers who support the students in the implementation of their projects were conducted before and after the project.
Evaluation of hazard prevention and damage after the heavy rain events in Saarland in May/June 2018
In May and June 2018, Saarland was hit by heavy rainfalls which caused high damages in a number of municipalities. As a consequence, the Saarland Ministry for the Environment and Consumer Protection (MUV) commissioned CEval GmbH to review the measures of hazard prevention and disaster management in the wake of these events. For this purpose, an analysis of strengths and weaknesses of hazard prevention and disaster management with regard to prevention, deployment and after-care was carried out in order to lay the foundation for identifying potential improvements and evidence-based recommendations. In addition, the damages caused by the heavy rainfalls should be assessed.
To analyse the strengths and weaknesses of hazard prevention and disaster management, 62 guided interviews were conducted with representatives of the Ministry of Interior, Construction and Sport, fire-fighting coordinators, local aid organisations and mayors of the most affected municipalities. The results of the content analysis of these interviews formed the basis for identifying potential for improvement and deriving concrete recommendations for action. The evaluation of damages in the area of municipal infrastructure, agriculture and private households was carried out quantitatively on the basis of the data provided by the client.
Development of an operational guidebook for integrating and using impact evaluation-based M&E approaches in the field of climate change adaption
Climate change adaption projects require methodologically elaborated Monitoring and Evaluation (M&E) Systems that are not limited to measuring Project Outputs but also Need to include outcomes and further Impacts as well as their attribution to the measures undertaken.
GIZ engaged the Center for Evaluation (CEval) for technical support to develop an operationally viable impact evaluation and provided examples for their applicability in adaptation projects. Results were presented in a guidebook for integrating and using impact evaluation-based M&E approaches in the field of climate change Adaptation. Target Groups of the Framework are ministries, donors, and other related institutions that are looking for the underlying Drivers of Adaptation and evidence-based findings to improve the design of future climate change adaptation projects.
Evaluation of the Projects “Sustainable Municipal Development through Partnership Projects” and “Support Program for Municipal Climate Protection and Climate Change Adaption Projects”
As part of Engagement Global – Service für Entwicklungsinitiativen (Service for Development Initiatives) the Service Agency Communities in One World in Germany ist the competence Center for local development cooperation. Founded in 2001 the Service Agency offers a platform for dialogue for municipalities in Germany and the South and supports development activities of local actors by providing them with Training, Information and advisory Services.
The two programs “Sustainable Municipal Development through Partnership Projects – Nachhaltige Kommunalentwicklung durch Partnerschaftsprojekte (Nakopa)” and “Support Program for Municipal Climate Protection and Climate Change Adaption Projects – Förderprogramm für kommunale Klimaschutz- und Klimaanpassungsprojekte (FKKP)” are core activities of the Service Agency. While Nakopa supports existing and new municipal partnership Projects in selected relevant fields of development cooperation, FKKP explicitly addresses municipal activities in the field of climate protection and climate change adaptation.
After the first Project cycle of three years has been completed, the current Evaluation aims at evaluating These two programs with respect to processual aspects of project Administration as well as Impacts of both the funded partnership Projects and the Support programs Nakopa and FKKP. Leading questions of this evaluation are guided by the OECD-DAC evaluation criteria and had been answered using the approach of contribution Analysis and documentary and empirical data. Hence, the evaluation was based on Monitoring data and Project documents on the one Hand and empirical data gathered through extensive partly standardized interviews and partly standardized online surweys with participating communities on the other hand.
Evaluation of the website www.bne-portal.de
The German Commission for UNESCO is a Quango in the field of multilateral cultural, educational, Research, and media policy. The website www.bne-portal.de is one of their efforts in contributing to the Global Action Programme on Education for Sustainable Development. The website is considered to open the field of education for sustainable development (German: Weltaktionsprogramm Bildung für nachhaltige Entwicklung) to actual as well as future protagonists in Germany. After a re-launch in 2016 and an uptime of one year, CEval GmbH was entrusted to evaluate the website www.bne-portal.de.
The evaluation of the website was focused on (a) clarity, relevance, and appeal of the provided information, (b) attractiveness of design and structure, (c) information accessibility, (d) practical relevance of information, (e) usefulness of provided contact channels, (f) degree of fulfilling user expectations, (g) intensity of use, (h) popularity/attractiveness of its education materials, and (h) popularity/attractiveness of its protagonist database.
The evaluation was based on an online user survey followed by quantitative data analysis.
