Impact analysis of the project “Kita im Aufbruch: Prozessbegleitung Richtung Nachhaltigkeit” [Kita im Aufbruch: Process support for sustainability in education]
The State Association for the Protection of Birds in Bavaria e.V. (Landesbund Vogelschutz, LBV) had implemented the environmental education project “Kita im Aufbruch: Process support for sustainability in education” between November 2020 and March 2022. The aim of the project was to contribute to the structural embedment of Education for Sustainable Development (ESD) in early childhood care by process support for selected Bavarian day-care centers. The CEval GmbH conducted an impact analysis of the project. By means of a before-and-after survey of day-care center staff, parents and day-care center operators, the impact of the project on the competencies of the pedagogical staff and the quality of educational offers as well as pedagogical processes in terms of ESD had been investigated. In addition, the CEval team further analyzed to what extent the project contributes to a holistic transformation of the learning and teaching environment towards a structural embedment of ESD in the daycare centers. In addition to pedagogical concepts, this also includes the areas of sustainable procurement and catering in the sense of a sustainable menu. In order to be able to clearly assign observed effects and impacts to the interventions of the project, the impact analysis was based on a quasi-experimental design including comparison groups (daycare centers without participation in process support) into the survey. The team combined, qualitative and quantitative survey methods in the process of data collection.
Evaluation of APPEAR projects (programme periods I and II)
The Austrian Partnership Programme in Higher Education and Research for Development (APPEAR) is conceived, guided and financed by the Austrian Development Agency (ADA) to implement its “Higher Education and Scientific Cooperation” strategy in support of Higher Education and Research for Development on an academic institutional level. The funded projects are implemented by the Austrian Agency for International Mobility and Cooperation in Education, Science and Research (OeAD GmbH). APPEARs overall objective is to strengthen the institutional capacities in higher education, research and management through Academic Partnerships with Austrian higher education institutions and through scholarships. The partnerships aim to improve the general standards in higher education, research and management by means of knowledge sharing and designing innovative projects. The achievement of these objectives should finally contribute to effective and sustainable reduction of poverty in the partner countries.
The overarching goal of the evaluation was to assess the ongoing and completed projects of the APPEAR periods I and II. Thereby, it provided insights about the projects adherence to the APPEAR guidelines and basic principles during their implementation, and about the extent to which they were to achieve their objectives and results. Furthermore, the projects were assessed according to the OECD/DAC criteria Relevance, Effectiveness, Efficiency, Impact and Sustainability. The objective of the evaluation was to identify the strengths, weaknesses and challenges of the APPEAR projects and present conclusions, lessons learnt and recommendations for the remaining or future projects.
The evaluation made use of a variety of methods. Primary data was collected during project visits to Ethiopia and Uganda, interviews and focus group discussions with project partners in Austria as well as virtual interviews with staff and key stakeholders of two projects in Nicaragua. The data collection instruments also comprised semi-standardized surveys, programme and project documentations and previous programme evaluations.
Evaluation of CAPAQUA
CAPAQUA (Development of Educational and Research CAPacity in Eastern Africa for the Sustainable Management of AQUAatic Ecosystems) aims at enhancing the capacity of Eastern African Higher Education, Science and Technology (HEST) institutions. CAPAQUA is jointly implemented by the University of Natural Resources and Life Sciences (BOKU), Vienna, the IHE Delft Institute for Water Education, the Egerton University (EGU), the Addis Ababa University (AAU), the Bahir Dar University and the Ethiopian Institute for Agricultural Research, National Fish and Other Aquatic Lives Research Center (EIAR-NFALRC). CAPAQUAs overall objective is to foster sustainable management of aquatic ecosystems and their resources in order to improve livelihoods in Eastern Africa and to contribute towards the achievement of the Sustainable Development Goals.
The main purpose of the evaluation was to identify successes as well as weaknesses of the project, and to provide recommendations for the project partners to improve the relevance, effectiveness, efficiency and sustainability of possible future CAPAQUA projects. The evaluation was also analysing the extent to which the cross-cutting issue gender mainstreaming was applied and identified the major factors influencing the achievement or non-achievement of the expected project results and objectives. In order to assess the project, the evaluation focussed on the project period from October 2015 to September 2018.
A rich set of mainly qualitative data was collected by means of in-depth interviews and focus group discussions with different stakeholders, programme participants, graduates as well as staff. Programme and project documentations provided secondary data and was complemented by primary data collected during focus group discussions. The geographic focus of the evaluation was the Eastern African region, with emphasis on Uganda, Kenya and Ethiopia, where empirical data was collected during short field visits. Additionally, Austrian institutions were visited and stakeholders interviewed.
Evaluation of the program “DLR-DAAD Research Fellowships”
Since 2008 the German Aerospace Center (DLR) and the German AcademicExchange Service (DAAD) are jointly implementing a research fellowship programme with a focus on energy, aeronautics, space and transport related disciplines. The programme aims at contributing to the internationalization of DLR institutes as well as promoting researchers who maintain a relation to Germany as partners and friends. The goal of the evaluation was twofold: On the one hand, it should assess the programmes medium- and long-term effects. On the other hand, it identified potentials for improvement and come up with suggestions for necessary adaptions in view of a possible future continuation of the programme. In terms of methodology, the evaluation relied on a mix of qualitative and quantitative data collection and analysis methods. Among others, guided interviews, focus group discussions, on-site visits as well as an online survey were implemented.
Scientific support of an impact analysis of the project “egal war gestern – towards sustainability at vocational schools”
The Landesbund für Vogelschutz in Bayern (LBV) e.V. implemented the project Towards sustainability at vocational schools in the school term 2018/19 for the second time. It aimed at mobilizing the youth for matters of sustainability. More precisely, the project was geared towards students at Bavarian vocational schools who find themselves in the important period of professional orientation. The goal was to direct their attention to sustainability questions and to strengthen their awareness of those issues. To achieve this goal, the students were expected to independently implement projects in their respective schools in order to contribute to a more sustainable school life.
The project was subject to an impact analysis, which was jointly implemented by LBV and CEval GmbH within the context of an accompanying evaluation. The purpose was to determine whether the intended effects of the project were actually achieved, that is, whether the attitudes and behavior of the participating students actually changed in relation to sustainability matters and whether those matters are more readily integrated into the everyday life at Bavarian vocational schools. In terms of methodology the analysis was based on a standardized survey before and after the project with both the participating students and a comparison group of students. Furthermore, guided interviews with the teachers who support the students in the implementation of their projects were conducted before and after the project.
Accompanying evaluation of the support program of the AMASE Masters-program
Together with a consortium of three other European universities (Luleå Technical University, Sweden, Barcelona Tech, Spain and Université de Lorraine, France) the European School of Materials (EUSMAT) at Saarland University offers the international Masters programme AMASE (Advanced Materials Science and Engineering). Its students have the opportunity to participate in an Integration Week in Saarbrücken right at the beginning of their studies, which aims at providing an orientation with regards to the course of the AMASE programme as well as strengthening the students? soft skills through a number of trainings. After the first year of study in the two-year AMASE programme, the students furthermore have the opportunity to take part in a Professional Summer School aiming at facilitating their career entry.
Both events are part of the AMASE support programme and were evaluated within the context of an accompanying evaluation. The main goal was to determine the use of those events for their respective target groups. In methodological terms two standardized surveys as well as a number of focus group discussions were conducted with participants of the two events.
Integration of the Programme “Service Agency Communities in One World” (SKEW) in the M&E-System of Engagement Global
Engagement Global gGmbH was established in 2012 as a central institution to promote and coordinate the development policy commitment of civil society, municipal and private sector actors as well as for development policy information and education work. SKEW a program of Engagement Global focuses in particular on the promotion of municipal development cooperation. In order to be able to monitor both the appropriate use of the funds and above all their effectiveness, Engagement Global has created a comprehensive monitoring and evaluation (M&E) system for all departments and programs. Since SKEW was excluded in the development of the monitoring system at that time, CEval and ISPO were commissioned to integrate SKEW into the organization-wide monitoring system.
As with the design of the company-wide M&E system, first the theory of change and indicators were reviewed or developed. Then the objectives and indicators were integrated into the strategical claims of Engagement Global. In order to complete the creation of a monitoring system for the SKEW, a data collection plan were created and the associated data collection instruments had been (further) developed.
Formative Evaluation of the Out-of-School-Children Initiative (OOSCI)
The focus of this evaluation was the Out-of-School-Children Initiative (OOSCI) by UNICEF and UNESCO Institute for Statistics (UIS). The initiative was launched in 2010 to achieve a reduction in the number of children out of school in participating countries. Means to achieve this goal were to create comprehensive profiles of out-of-School children in each Country, identify the barriers that push children out of school and to propose changes in the policies and strategies of the partner country to address these barriers.
The evaluation aimed at testing the validity of the theory of change and its assumptions. It is used to strengthen the program logic and to provide a formative assessment of the progress towards the overall goal of achieving a reduction in out-of-school children. The evaluation also provides an in-depth insight into benefits for the different stakeholders and their involvement in the initiative. Furthermore, the findings of the evaluation are used to meet the accountability requirements by the donors of OOSCI.
Main source of information of this evaluation have been documents like OOSCI country studies, OOSCI regional studies, government plans and documents from other donors (e.g. the World Bank, UNESCO, foreign donors etc.). In a document review we assessed progress made through OOSCI for a sample of 42 countries and assessed the contents within the document using a 4-step coding. These codings were used for a descriptive analysis and additionally for a Qualitative Comparative Analysis (QCA). QCA enabled us to analyze thoroughly which set of conditions leads to which outcome. Further instrument of data collection has been an online survey to all country offices. Additional qualitative data has been collected in the US and during eight country visits. The evaluation has been conducted together with Proman and further Consultants. CEval took over the lead with regard to the evaluation methodology.
Evaluation of the demographic monitoring tool demowanda: Demand-oriented coaching on self-evaluation
“The BAuA explores the conditions and developments in gainful employment with a view to providing workers with the best possible protection and Support both today and in the future.” (s. BAuA-Website). Leading the project Demografischer Wandel in der Arbeitswelt (demowanda) is one of BAuAs Tasks. demowanda is a demographic monitoring tool, which has been ordered by the Federal Ministry of Labour and Social Affairs.
BAuA realized a self-evaluate of the demowanda project – their collaboration project with BiB, BIBB, DZA, IAB and RKI. The objective of BAuA’s self-evaluation was the analysis and evaluation of the projects implementation process, its outputs, outcomes and impacts.
In order to successfully realize the self-evaluation, BAuA commissioned CEval GmbH to accompany their self-evaluation and to provide them with demand-oriented coaching/training. Employees of BAuA, who are involved in the self-evaluation of demowanda, were enabled to extend their knowledge on self-evaluation. Moreover, coaching/training ensured the conceptual as well as methodological quality of the self-evaluation project.
The demand-oriented coaching included phone and on-site consulting, inhouse trainings and workshops, and support of development evaluation materials/documents.
Process Evaluation of the career support program ‘Securing Talents – Shaping the Future’ (2017-2018)
Process Evaluation of the career support program ‘Securing Talents – Shaping the Future’ (2017/2018)
The program ‘Securing talents – shaping the future’ by the Cusanuswerk aims to support future female leaders. The program focuses on mentoring elements that is tutorial support for scholarship holders by professionally experienced academics. It also focuses on occupationally specific training and support measures as well as on building a peer-network. The CEval has been conducting accompanying external evaluations to ensure quality management since the start of the career support program in 2007. An accompanying evaluation took also place during the sixth program run 2017/18. The main task of this evaluation was to provide the responsible employees with management-related information. In this context, questions regarding satisfaction with the offered events, quality of tutorial relationships as well as questions regarding professional and personal use of the relationships for both sides were especially relevant. In regards to methods, the evaluation covers standardized questionnaires of mentees and mentors compared at three points in time.