International Program for Development Evaluation Training (IPDET)
The International Program for Development Evaluation Training (IPDET) is an executive training program in development evaluation. From 2018 to 2024, it was managed jointly by the Center for Continuing Education at the University of Bern (Switzerland), the Center for Evaluation at Saarland University (Germany), and the Independent Evaluation Group of the World Bank. Since 2021, IPDET has been a key implementing partner of the Global Evaluation Initiative (GEI).
IPDET aims to provide managers and practitioners in the field of evaluation with the tools required to evaluate policies, programs and projects at local, regional, national, and global levels, as well as to commission, manage, and especially use those evaluations for decision-making. Further information on IPDET can be found on the website: https://ipdet.org/.
Pilot project for the joint match day follow-up in German professional football
The Daniel Nivel Foundation aims to improve the relationship between football fans and the police. To that end, the foundation cooperates with national football associations and in 2018 has released funds for two pilot projects, one with the French Football Federation (FFF) and one with the German Football Association (DFB). On the German side, the following idea was developed within the so called future workshops: to establish a joint match day follow-up, to create a framework for dialogue between football fans and police. In this way, conflicts would be dealt with after match days and solutions for future events could be developed.
The DFB had commissioned CEval GmbH to provide scientific support for this project. The tasks of the CEval GmbH ranged from the conception of the project (follow-up process, selection of pilot locations, etc.) to the development of suitable instruments for fan surveys and suiting formats for match day follow-ups, as well as the support of the actors in the practical implementation and moderation of the follow-up process. Therefore, different procedures and formats for the match day follow-ups had been tested, in order to present an applicable concept after the pilot phase. All steps had been carried out in close cooperation with the DFB project team. There was also a regular, project related exchange with the Fan Projects Coordination Office (KOS) and the German Football League (DFL). The project was scheduled to run from January 2019 to May 2020. In the first half of 2019, a detailed project conception and elaboration took place, including workshops with all participating actors. In the second half of the year, during the first half of the 2019/2020 season, the developed concept for the match day follow-up had been implemented and tested in cooperation with selected pilot locations from professional football. Central partners had been the Fan Projects of the respective locations. This was followed by an evaluation and, if necessary, adaptation of the matchday follow-up concept and documentation of the project in the form of a final report.
Evaluation of the project “Digital Leadership” of the Führungsakademie Baden-Württemberg
Since March 2018, the Führungsakademie Baden-Württemberg (FüAK) has been offering the “Digital Leadership” programme. The programme comprises various advanced training formats to promote digital skills for managers in the administrative bodies of the federal state of Baden-Württemberg. The aim is to prepare these managers for the requirements of digital shift, to network them with one another and to equip them with the necessary skills for driving and shaping digital change in their institutions. The program’s formats include professional training (e.g. programming, design thinking), networking events, learning journeys to innovative companies and topic-specific focus groups. The evaluation focused on the different training formats of the programme. These were evaluated in regard to their effects. The focus was on the outcome level, especially on the short- and medium-term effects of the different events. In the sense of the participatory evaluation approach of CEval GmbH, the theory of change was developed together with the project team. The effects were assessed within the evaluation using a multi-method approach. On the one hand, participant questionnaires were developed and used at the end of an event and after a defined impact phase. On the other hand, interviews were conducted with different stakeholder groups in order to obtain more in-depth assessments of relevant aspects of the programme. In this way, the advantages of both data collection methods were combined and used to evaluate the training formats.
Development and Implementation of a M&E-System for the ‘EinTanzHaus’
The ‘EinTanzHaus’ is expected to be a center for contemporary dance in a cultural monument of particular importance in the City of Mannheim. The EinTanzHaus team and its director Daria Holme have jointly created the preconditions to develop a dance center in Southern Germany with the exemplary conversion of a protected historic church building, which can open up new perspectives for the dance scene in Germany due to its infrastructure, its stage size, its central location and the artistic experience.
The task of CEval GmbH was to develop and implement a M&E concept that serves for the management of the project and the regular review of goal attainment and impact assessment. Therefore, a target system and value-added indicators for the ‘EinTanzHaus’ as well as a data collection plan were developed together with the stakeholders in a first step. Further, CEval GmbH was involved in all steps of the data collection, evaluation and interpretation and thus supports the parties involved in the strategic management of the project and the evaluation of the projects goal achievement.
Data collection for the certificate “Fan project in accordance with the National Plan for Safety in Sports”
In Germany, there is a dense network of social work institutions in the field of adolescent soccer fans (fan projects) that work in accordance with the National Plan for Safety in Sports (German: Nationales Konzept Sport und Sicherheit). A certification of those fan projects has been launched in 2010. Offering fan projects an instrument of quality control, the certification is intended to pursue multiple goals: (a) establishing a common standard for fan projects, (b) increasing confidence in the work of fan projects, and (c) improving the working context of fan projects. The certificate has been awarded by the Quality Management Group of the Centre for Fan Project Coordination of the German Sports Youth (German: Koordinationsstelle Fanprojekte bei der Deutschen Sportjugend). Members of the Quality Management Group belong to several stakeholder groups: soccer, federal authorities (for family affairs), science, and fan projects themselves.
During a first subproject (5/2015-6/2016), CEval GmbH (a) revised the certification process, (b) tested the revised certification process on five fan projects, and (c) developed a standardized, feasible concept for collecting information necessary for certification. Thereby the certificate attests fan projects high quality of work.
Within second subproject, CEval GmbH supported the nationwide roll-out of the revised certification by collecting all necessary information on all the remaining German soccer fan projects.
Impact analysis of the project “Kita im Aufbruch: Prozessbegleitung Richtung Nachhaltigkeit” [Kita im Aufbruch: Process support for sustainability in education]
The State Association for the Protection of Birds in Bavaria e.V. (Landesbund Vogelschutz, LBV) had implemented the environmental education project “Kita im Aufbruch: Process support for sustainability in education” between November 2020 and March 2022. The aim of the project was to contribute to the structural embedment of Education for Sustainable Development (ESD) in early childhood care by process support for selected Bavarian day-care centers. The CEval GmbH conducted an impact analysis of the project. By means of a before-and-after survey of day-care center staff, parents and day-care center operators, the impact of the project on the competencies of the pedagogical staff and the quality of educational offers as well as pedagogical processes in terms of ESD had been investigated. In addition, the CEval team further analyzed to what extent the project contributes to a holistic transformation of the learning and teaching environment towards a structural embedment of ESD in the daycare centers. In addition to pedagogical concepts, this also includes the areas of sustainable procurement and catering in the sense of a sustainable menu. In order to be able to clearly assign observed effects and impacts to the interventions of the project, the impact analysis was based on a quasi-experimental design including comparison groups (daycare centers without participation in process support) into the survey. The team combined, qualitative and quantitative survey methods in the process of data collection.
Evaluation of APPEAR projects (programme periods I and II)
The Austrian Partnership Programme in Higher Education and Research for Development (APPEAR) is conceived, guided and financed by the Austrian Development Agency (ADA) to implement its “Higher Education and Scientific Cooperation” strategy in support of Higher Education and Research for Development on an academic institutional level. The funded projects are implemented by the Austrian Agency for International Mobility and Cooperation in Education, Science and Research (OeAD GmbH). APPEARs overall objective is to strengthen the institutional capacities in higher education, research and management through Academic Partnerships with Austrian higher education institutions and through scholarships. The partnerships aim to improve the general standards in higher education, research and management by means of knowledge sharing and designing innovative projects. The achievement of these objectives should finally contribute to effective and sustainable reduction of poverty in the partner countries.
The overarching goal of the evaluation was to assess the ongoing and completed projects of the APPEAR periods I and II. Thereby, it provided insights about the projects adherence to the APPEAR guidelines and basic principles during their implementation, and about the extent to which they were to achieve their objectives and results. Furthermore, the projects were assessed according to the OECD/DAC criteria Relevance, Effectiveness, Efficiency, Impact and Sustainability. The objective of the evaluation was to identify the strengths, weaknesses and challenges of the APPEAR projects and present conclusions, lessons learnt and recommendations for the remaining or future projects.
The evaluation made use of a variety of methods. Primary data was collected during project visits to Ethiopia and Uganda, interviews and focus group discussions with project partners in Austria as well as virtual interviews with staff and key stakeholders of two projects in Nicaragua. The data collection instruments also comprised semi-standardized surveys, programme and project documentations and previous programme evaluations.
Evaluation of CAPAQUA
CAPAQUA (Development of Educational and Research CAPacity in Eastern Africa for the Sustainable Management of AQUAatic Ecosystems) aims at enhancing the capacity of Eastern African Higher Education, Science and Technology (HEST) institutions. CAPAQUA is jointly implemented by the University of Natural Resources and Life Sciences (BOKU), Vienna, the IHE Delft Institute for Water Education, the Egerton University (EGU), the Addis Ababa University (AAU), the Bahir Dar University and the Ethiopian Institute for Agricultural Research, National Fish and Other Aquatic Lives Research Center (EIAR-NFALRC). CAPAQUAs overall objective is to foster sustainable management of aquatic ecosystems and their resources in order to improve livelihoods in Eastern Africa and to contribute towards the achievement of the Sustainable Development Goals.
The main purpose of the evaluation was to identify successes as well as weaknesses of the project, and to provide recommendations for the project partners to improve the relevance, effectiveness, efficiency and sustainability of possible future CAPAQUA projects. The evaluation was also analysing the extent to which the cross-cutting issue gender mainstreaming was applied and identified the major factors influencing the achievement or non-achievement of the expected project results and objectives. In order to assess the project, the evaluation focussed on the project period from October 2015 to September 2018.
A rich set of mainly qualitative data was collected by means of in-depth interviews and focus group discussions with different stakeholders, programme participants, graduates as well as staff. Programme and project documentations provided secondary data and was complemented by primary data collected during focus group discussions. The geographic focus of the evaluation was the Eastern African region, with emphasis on Uganda, Kenya and Ethiopia, where empirical data was collected during short field visits. Additionally, Austrian institutions were visited and stakeholders interviewed.
Evaluation of the program “DLR-DAAD Research Fellowships”
Since 2008 the German Aerospace Center (DLR) and the German AcademicExchange Service (DAAD) are jointly implementing a research fellowship programme with a focus on energy, aeronautics, space and transport related disciplines. The programme aims at contributing to the internationalization of DLR institutes as well as promoting researchers who maintain a relation to Germany as partners and friends. The goal of the evaluation was twofold: On the one hand, it should assess the programmes medium- and long-term effects. On the other hand, it identified potentials for improvement and come up with suggestions for necessary adaptions in view of a possible future continuation of the programme. In terms of methodology, the evaluation relied on a mix of qualitative and quantitative data collection and analysis methods. Among others, guided interviews, focus group discussions, on-site visits as well as an online survey were implemented.
Scientific support of an impact analysis of the project “egal war gestern – towards sustainability at vocational schools”
The Landesbund für Vogelschutz in Bayern (LBV) e.V. implemented the project Towards sustainability at vocational schools in the school term 2018/19 for the second time. It aimed at mobilizing the youth for matters of sustainability. More precisely, the project was geared towards students at Bavarian vocational schools who find themselves in the important period of professional orientation. The goal was to direct their attention to sustainability questions and to strengthen their awareness of those issues. To achieve this goal, the students were expected to independently implement projects in their respective schools in order to contribute to a more sustainable school life.
The project was subject to an impact analysis, which was jointly implemented by LBV and CEval GmbH within the context of an accompanying evaluation. The purpose was to determine whether the intended effects of the project were actually achieved, that is, whether the attitudes and behavior of the participating students actually changed in relation to sustainability matters and whether those matters are more readily integrated into the everyday life at Bavarian vocational schools. In terms of methodology the analysis was based on a standardized survey before and after the project with both the participating students and a comparison group of students. Furthermore, guided interviews with the teachers who support the students in the implementation of their projects were conducted before and after the project.