Evaluation and counselling of the project “Digital Civic Education and Counselling”
Against the background of the increasing digitalisation of all areas of society, the Friedrich-Ebert-Stiftung (FES), whose central task is political education and counselling, has launched the internal project “Digital Education and Counselling – FES digital”. Its aim is to anchor digital formats in the FES’s education and advisory services.
The FES had commissioned schmidt evaluation und CEval GmbH to conduct an accompanying evaluation of the project, which was to contribute to clarifying the project goals and indicators as well as those of the individual digital formats and to continuously monitor the achievement of goals.
Methodologically, the evaluation was based on the analysis of existing secondary data, on a benchmarking with regard to suitable target indicators and target values as well as on the implementation of a baseline, interim and final survey by means of online questionnaires and online self-assessments on the respective status of the digitalisation of the political education offers.
Evaluation of the prevention campaign “Mach Dein Passwort stark!”
In January 2019, the Ministry of the Interior of North Rhine-Westphalia (IM NRW) commissioned the NRW State Criminal Police Office (LKA NRW) to develop a prevention and publicity campaign to combat ‘cybercrime’. The aim of the campaign was to strengthen the population in NRW in the safe use of digital end devices and thereby reduce opportunities for cybercrime in the broader sense and make the work of the perpetrators more difficult. The cross-media campaign “Make your password strong!” was launched in October 2020.
CEval GmbH was commissioned to carry out an accompanying evaluation for the campaign with the two sub-areas of process and impact evaluation. The aim of the study was to evaluate the implementation process as well as the effectiveness of the “Make your password strong!” campaign and to derive conclusions and recommendations for future police prevention campaigns based on these results.
The evaluation was based on a broad mix of qualitative and quantitative data collection involving all actors involved in the design and implementation of the campaign as well as the target groups. The design also included an experimental trial set-up for comparative impact evaluation of the different components of the campaign. Descriptive and inferential statistical methods as well as qualitative content analyses were realised within the framework of the data analysis.
Evaluation of the programme “Mentoring Hessen – Frauen in Wissenschaft und Wirtschaft”
Mentoring Hessen is a joint project of Hessian universities and universities of applied sciences to advance the careers of women. The aim is to increase the percentage of women in attractive specialist and management positions in both academia and business. With the help of four different lines of support, young women are supported and guided through the decisive phases of their professional development from student to professor! The close cooperation with renowned companies and research institutions also opens up career opportunities outside of academia. Through mentoring, training and networking, young women are supported in the decisive phases of their professional development.
The goal of the evaluation was to analyse the implementation of the concept and organisational model of Mentoring Hessen in order to develop recommendations for the optimal further development of the programme based on the findings. The evaluation adopted two perspectives of analysis: 1. analysis of the conception, implementation and goal achievement of Mentoring Hessen: Evaluation of the programme conception and implementation, assessment of the achievement of objectives and effectiveness, derivation of recommendations for optimising the conception, implementation and achievement of objectives; 2. Analysis of the interaction of the stakeholders in the joint project Mentoring Hessen: Analysis of the structure and its functionality as well as evaluation of the interaction of the different groups of actors, derivation of recommendations for the optimisation of organisational structure and cooperation.
The data collection combined a document analysis with a comprehensive secondary analysis of existing data from target group surveys, guided interviews and group discussions. The data analysis combined descriptive statistics and qualitative content analysis.
Study program evaluations of the Theaterakademie August Everding
The August Everding Theatre Academy, Munich, is an academic teaching and learning theatre that offers eight courses of study in the field of stage professions at Bachelor’s and Master’s level. These are realised in cooperation with the four Munich universities and the three Munich state theatres.
The Theatre Academy has commissioned CEval GmbH to carry out evaluations of five study programmes. The aim is to collect an evaluation of the theoretical and practical orientation by the students of the various study programmes in the sense of quality development, in order to further optimise the study programmes on the basis of this data. For this purpose, the students are surveyed online using partially standardised questionnaires.
Employee survey on working time and mobile work
The TU Darmstadt (TUD) pursues the goal of developing working time models and regulations for mobile work that are oriented towards the needs of employees. In this way, the satisfaction of employees with the organisation of working time and place of work is to be promoted. To this end, the TUD Executive Board commissioned CEval GmbH to conduct an employee survey on the topics of working hours and mobile work.
The surveys were thematically oriented in such a way that, on the one hand, the lived practice of organising working time and using mobile work was recorded, and on the other hand, the respondents’ satisfaction with this practice as well as suggestions and wishes for the future. Without this being foreseeable when the study was initiated, the Corona pandemic led to considerable adjustments in the design of working conditions during the survey period, especially with regard to mobile work. During the fine-conception of the study, these changes were taken into account by also including questions on the (new) experiences regarding mobile working during the pandemic.
The data collection combined a document analysis with a partially standardised online survey of all employee groups at the TUD as well as guideline-based interviews. The data analysis combined descriptive statistics and qualitative content analysis.
Accompanying evaluation of the support programme “Female Producer Collective” (2022)
The support program “Female Producer Collective” of GOFORMUSIC gGmbH is a special support project for female producers, who receive coaching in the areas of “Creativity and Idea Generation”, “Recording & Sound Design”, “Mixing & Mastering” and “Business & Marketing”. The aim of the project is to specifically promote female producers, to counteract the low quota of women in the production sector and to inspire female musicians to produce. CEval GmbH was commissioned with the accompanying evaluation of the funding project. From a methodological point of view, focus group interviews and a short survey via a standardized questionnaire had been conducted for this purpose.
Process Evaluation of the career support program ‘Securing Talents – Shaping the Future’ (2021/2022)
The program ‘Securing Talents – Shaping the Future’ of the Cusanuswerk – Bischöfliche Studienförderung aimed to promote the next generation of female leaders. To this end, the program relies on mentoring elements, i.e. tutorial support for female scholarship holders of German scholarship programs for the gifted by academics with professional experience, training and support measures geared to specific professions, and the establishment of a peer network. Since the start of the career development program in 2007, CEval had conducted external evaluations to ensure the quality of the program. An accompanying evaluation took also place in the eighth program run in 2021/2022. The central task of this evaluation was to provide the Cusanuswerk staff responsible for implementation with timely information relevant to management. In accordance with the program’s objectives, questions regarding satisfaction with the events offered, the quality of the tutorial relationships, and the professional and personal benefits of the relationship for both sides are of particular interest. Methodologically, the evaluation is based on standardized online surveys of mentees and mentors at three points in time.
Process Evaluation of the support program “Collegium Cusanum”
The program ‘Collegium Cusanum’ of the Cusanuswerk – Bischöfliche Studienförderung is aimed at all scholarship holders of the Cusanuswerk (students and doctoral candidates) as well as Altcusaner who aspire to a professional activity in the school or pedagogical field or are already doing so. The support program thus makes a sustainable contribution to the personal development and qualification of teachers and educators. To this end, the program relies on mentoring elements, i.e. tutorial support for Cusanuswerk scholarship holders by teachers and educators with professional experience. The CEval carries out external evaluations accompanying the process for quality assurance purposes. The central task of this evaluation is to provide the Cusanuswerk staff responsible for implementation with timely information relevant to control. Of particular interest, in accordance with the objectives of the program, are questions regarding satisfaction with the events offered, the quality of tutorial relationships, and the professional and personal benefits of the relationship for both parties. Methodologically, the evaluation is based on standardized online surveys of mentees and mentors at three points in time.
Framework contract for the further development and implementation of systematic results-based approaches through monitoring and evaluation for DAAD programs (2020)
The DAAD understands itself as a modern science organization that strives to continuously improve its effectiveness through continuous learning and the willingness to change its work. This approach is formulated in the DAAD’s Integrated Monitoring and Evaluation Concept, which was adopted in 2019. The concept identifies institutional learning, quality assurance, results-based orientation, transparency in processes and legitimacy as important dimensions to which the various monitoring and evaluation activities are geared. Doing so, from an overarching perspective, the strategic orientation of the DAAD and its programs is also to be supported on an evidence-based basis.
The framework contract with CEval GmbH aims to meet the increasing demands for results-based orientation and control in the various funding areas of the DAAD even better by supporting the implementation of results-based orientation and control for existing and new programmes in the AA- and BMBF-funding areas and by accompanying the results-based orientation of newly designed BMZ programmes. To this end, services are provided in the following areas:
Evaluation of the Alexander von Humboldt Foundation’s Research Fellowship Programmes for Researchers from Germany (FLF, JSPS, MOST)
The Alexander von Humboldt Foundation (AvH) enables highly qualified academics from all over the world to spend longer periods of research in Germany and supports the resulting academic and cultural links between Germany and abroad. The Foundation’s network now comprises over 29,000 researchers from all disciplines in more than 140 countries.
To strengthen the Humboldt Network, the AvH’s programme portfolio also includes three research fellowship programmes that support researchers from Germany to work at research institutions abroad. The BMBF-funded Feodor Lynen Research Fellowship Programme (FLF) enables German researchers to spend a longer period of time researching abroad. In this way, the existing Humboldt Network abroad is strengthened and the potential of Humboldt alumni abroad is tapped in their function as academic advisors and “door openers” for (junior) researchers from Germany. For the target countries Japan and Taiwan, the Humboldt Foundation cooperates with the Japan Society for the Promotion of Science (JSPS) and the Ministry of Science and Technology, Taiwan (MOST) within the framework of the JSPS Programme and the MOST Programme. The AvH also offers support services for alumni from all three scholarship programmes from the BMBF funding. These include reintegration assistance, return scholarships, renewed research stays, institute partnerships as well as annual conferences and networking events.
The evaluation is intended to fulfil two functions: On the one hand, it should provide an evidence-based foundation for the further development of the programmes in the sense of a formative evaluation. On the other hand, it is to be used to account to the funding agencies and the public for the implementation and achievement of the goals of the scholarship programmes. The evaluation focuses on the extent to which the Humboldt Network has benefited from the three programmes and whether suitable research fellows have been selected with regard to the programme goals.
The evaluation is characterised by a participatory and multi-method approach. The evaluation combines document and secondary analyses with qualitative (guided intensive interviews) and quantitative survey procedures (partially standardised online surveys).
