Process Evaluation of the career support program ‘Securing Talents – Shaping the Future’ (2023/2024)
The program ‘Securing Talents – Shaping the Future’ of the Cusanuswerk – Bischoefliche Studienförderung aims to promote the next generation of female leaders. To this end, the program relies on mentoring elements, i.e. tutorial support for female scholarship holders of German scholarship programs for the gifted by academics with professional experience, training and support measures geared to specific professions, and the establishment of a peer network. Since the start of the career development program in 2007, CEval has conducted external evaluations to ensure the quality of the program. An accompanying evaluation also took place in the program run in 2023/2024.
The central task of this evaluation was to provide the Cusanuswerk staff responsible for implementation with timely information relevant to management. In accordance with the program’s objectives, questions regarding satisfaction with the events offered, the quality of the tutorial relationships, and the professional and personal benefits of the relationship for both sides were of particular interest. Methodologically, the evaluation was based on standardized baseline and final online surveys of mentees and mentors.
Perspective study on the DAAD program “HAW.International”
The German Academic Exchange Service (DAAD) is the world’s largest funding organization for the international exchange of students and academics. In addition to awarding scholarships, the DAAD also promotes the internationalization of German universities. The DAAD funding program ‘HAW.International’ has been offered since 2019 and is aimed exclusively at universities of applied sciences (HAW). As part of the program, students and lecturers at HAW are supported with individual scholarships as well as structure-building and model and cooperation projects. In the long term, the program is intended to bring about a comprehensive and sustainable internationalization of HAW and promote its strategic positioning as an innovative university model for practice-oriented academic education worldwide.
In preparation for a further funding phase from 2025, the perspective study summarised the results of the funding to date and examined the need for further support at HAW to strengthen their internationality and international visibility.
The study was designed as a monitoring-based secondary data study and therefore based its analysis primarily on the qualitative and quantitative monitoring data on the program available at the DAAD. In addition, qualitative individual and group interviews were also utilized. The results should provide the DAAD with a basis for planning and designing the second program generation of HAW.International.
Framework contract for the further development and implementation of systematic results-based approaches through monitoring and evaluation for DAAD programs (2023)
The DAAD understands itself as a modern science organization that strives to continuously improve its effectiveness through continuous learning and the willingness to change its work. This approach is formulated in the DAAD’s Integrated Monitoring and Evaluation Concept, which was adopted in 2019. The concept identifies institutional learning, quality assurance, results-based orientation, transparency in processes and legitimacy as important dimensions to which the various monitoring and evaluation activities are geared. Doing so, from an overarching perspective, the strategic orientation of the DAAD and its programs is also to be supported on an evidence-based basis.
The framework agreement focuses on impact-oriented program planning, impact-oriented monitoring and target achievement and thus on further development and learning processes as well as on reporting obligations within the framework of performance monitoring. However, it also focuses on the effective and efficient use of the data generated by the Results-oriented Monitoring (RoM) in the context of monitoring-based analyses and evaluations and can thus make a significant contribution to strengthening and increasing the benefits of the RoM.
For this purpose, services are provided in the following areas:
– Development of results frameworks and indicator catalogues
– Development of program-specific monitoring instruments and revision of the program documents according to the requirements of the monitoring concept
– Data processing and analysis
– Monitoring based analyses and evaluations
– Internal and external workshops for the implementation of RoM
– Support in the context of monitoring visits to project partners
– Consultations on further approaches and methods in the field of monitoring and evaluation
– Further consulting and training services
Support for the evaluation of the Azubi Kolleg Lübeck
The Lübeck Trainee Kolleg is a programme of the Joachim Herz Foundation and the Lübeck Possehl Foundation, which supports 25 Lübeck trainees over two years in their personal development and promotes social skills. Seminars, workshops and training sessions are designed to enable the trainees to recognise and develop their strengths and potential and to increase their self-confidence, sense of responsibility and social commitment. In the block events, further training is offered on three major thematic blocks: a) communication, b) society/ politics/ economy and c) cultural education. Further goals of the programme are the exchange between the trainees, the joint learning process and the development of a network. In the long term, the programme is intended to contribute to the personal development and professional establishment of the trainees and to increase the attractiveness of vocational training. In addition, it is intended to give the development of talented trainees a greater presence in society. As a pilot project, the Azubi Kolleg is designed to run for five years.
In 2023, CEval GmbH was assigned again to support the evaluation of the programme. On the basis of the results-based framework and indicator catalogue developed in the previous project, the baseline data of the second cohort had been analyzed in 2023 in comparison with the data of the first cohort. In addition, online surveys among trainees, teachers and supervisors as well as focus group interviews with trainees and guided interviews with teachers and supervisors at the end of the programme are planned as the basis for a before-and-after comparison had been conducted.
Ex-ante survey in the programme “Azubi Kolleg Lübeck”
The Lübeck Trainee Kolleg is a programme of the Joachim Herz Foundation and the Lübeck Possehl Foundation, which supports 25 Lübeck trainees over two years in their personal development and promotes social skills. Seminars, workshops and training sessions are designed to enable the trainees to recognise and develop their strengths and potential and to increase their self-confidence, sense of responsibility and social commitment. In the block events, further training is offered on three major thematic blocks: a) communication, b) society/ politics/ economy and c) cultural education. Further goals of the programme are the exchange between the trainees, the joint learning process and the development of a network. In the long term, the programme is intended to contribute to the personal development and professional establishment of the trainees and to increase the attractiveness of vocational training. In addition, it is intended to give the development of talented trainees a greater presence in society. As a pilot project, the Azubi Kolleg is designed to run for five years.
In order to continuously improve the programme, CEval GmbH was commissioned to strengthen the impact orientation of the programme, therefore, in a first step, CEval GmbH developed a results-based framework and an indicator catalogue. On this basis, the next step was to conduct ex-ante surveys in the form of online surveys among the trainees and among the persons who supervise the trainees in the company (e.g. as a trainer, management, personnel department, etc.) or at school (e.g. as a class teacher, head of department, etc.). On the other hand, the surveys were intended to provide information about the trainees’ self-assessment and others’ assessment of their skills, personality traits, leisure activities or fields of interest at the beginning of their participation in the programme.
Accompanying evaluation of the support programme “Female Producer Collective“ (2023)
The program “Female Producer Collective” of GOFORMUSIC gGmbH is a special support project for female producers, who receive coaching in the areas of “Creativity and Idea Generation”, “Recording & Sound Design”, “Mixing & Mastering” and “Business & Marketing”. The aim of the project is to specifically promote female producers, to counteract the low quota of women in the production sector and to inspire female musicians to produce. CEval GmbH was already commissioned with the accompanying evaluation of the funding project in the first funding round (2022). In 2023, the programme “Female Producer Collective” started with a second round, whereby CEval GmbH had again conducted an accompanying evaluation. From a methodological point of view, focus group interviews with participants and coaches of the programme as well as an online survey via a standardised questionnaire had been conducted.
Data collection for the certificate “Fan project in accordance with the National Plan for Safety in Sports” (German: „Fanprojekt nach dem Nationalen Konzept Sport und Sicherheit“)
In Germany, there is a dense network of social work institutions in the field of adolescent soccer fans (“fan projects”) that work in accordance with the National Plan for Safety in Sports (German: Nationales Konzept Sport und Sicherheit). A certification of those fan projects has been launched in 2010. Offering fan projects an instrument of quality control, the certification is intended to pursue multiple goals: Establishing a common standard for fan projects, increasing confidence in the work of fan projects, and improving the working context of fan projects. The certificate is awarded on the basis of a recommendation by the Quality Management Group of the Centre for Fan Project Coordination of the German Sports Youth (German: Koordinationsstelle Fanprojekte bei der Deutschen Sportjugend). The Quality Management Group includes representatives from soccer, the highest state and family authorities, science, and fan projects themselves. After the certification was revised and tested by CEval GmbH in two previous projects and subsequently rolled out nationwide by collecting the information required for the certification in all fan projects, CEval GmbH has now been commissioned with the data collection for the third phase of the certification process for the current 71 fan projects.
Evaluation of a digital Senior Citizens Portal “digital.vital”
The senior citizens’ portal “digital.vital” is a model project of the Amt Hüttener Berge, the district of Germersheim and the city of Püttlingen. The aim of the portal is to make a contribution to meeting the challenges associated with demographic change and advancing digitalisation: Through needs-based offers, people are to be supported in the areas of nutrition, exercise and leisure activities/social participation (incl. housing, mobility, support) in achieving and maintaining a high quality of life in old age. At the same time, the portal also aims to strengthen the digital participation of older people. The project started in July 2020 and from autumn 2022 various services and information will be made available on the portal.
The Amt Hüttener Berge has commissioned CEval GmbH to evaluate the senior citizens’ portal in order to gain insights into the quality of the information and services offered and to identify possible potential for optimising the user experience in terms of portal functionality and content. Methodologically, a total of four participant observations will be carried out for the structured evaluation of the user experience among the pilot users in the context of so-called digital workshops at various locations. The participant observations will be supplemented by follow-up interviews with selected participants a few days after their visit to the digital workshops in order to collect data on the actual benefits of the workshops and the use of the website by the participants. In addition, a standardised online survey will be conducted among senior citizens’ organisations and actors in the partner municipalities on their perceptions and professional assessments of the senior citizens’ portal, its services and potential for further development.
Evaluation Lectureships
As the world’s largest funding organization for the exchange of students and academics, the DAAD promotes teaching stays for academics at universities abroad with its Lectureships Abroad programs. With its lectureships abroad, the DAAD pursues foreign cultural policy, development policy, higher education policy and program policy objectives. Depending on the host country, the host institution and the nature of the lectureship, these objectives vary in their focus.
The evaluation focused on the long-term, short-term, and Herder lectureships in the period under review from 2012 to 2022.
The evaluation pursued three central objectives: To examine whether the programs in their current design are up-to-date and needs-oriented, to support the accountability of the DAAD to the Federal Foreign Office (Auswärtiges Amt), and to support the strategic positioning of the programs through their further development.
In terms of methodology, a mix of qualitative and quantitative survey and evaluation methods were combined. Among other things, a comprehensive document and secondary data analysis, virtual site visits/field study, guided interviews, group discussions and online surveys were carried out. Examination and revision of the impact model as well as indicator development were part of the evaluation.
Evaluation of the German Colombian Peace Institute – CAPAZ
The German Colombian Peace Institute (CAPAZ) is one of five Centers of Excellence for Research and Teaching abroad, which has been financed by the German Federal Foreign Office since 2016. The institute is considered an academic platform for cooperation which facilitates the exchange of knowledge and experience in the field of peace-related research between research, teaching, civil society and state institutions, and, in doing so, intends to contribute to the establishment of a sustainable peace order in Colombia. The fields of activity of the CAPAZ Institute encompass teaching, research, policy advice, and knowledge transfer.
With the current funding period coming to an end and in view of an envisaged next funding period, the German Academic Exchange Service (DAAD) mandated CEval GmbH with an evaluation of the CAPAZ Institute. The mission of the evaluation was to review the project’s progress and results and, on that basis, put forth recommendations allowing to further develop and possibly improve the project and its funding. In that respect, the evaluation was based on the following evaluation criteria: relevance, effectiveness, efficiency, sustainability, as well as coordination, complementarity, and coherence.
The methodological approach chosen for this evaluation is theory-based and participatory. Furthermore, it is based on a combination of primarily qualitative methods of data collection and analysis, including an analysis of documents and secondary data, exploratory interviews as well as in-depth guided interviews and focus group discussions.
