Alumni survey of the European School of Materials (EUSMAT)
The European School of Materials (EUSMAT) at Saarland University offers various international study programs at Bachelors, Masters and PhD level. In order to stay in touch with the alumni of these programs, EUSMAT is currently expanding its alumni activities.
To verify how well the current alumni activities fit the needs and to identify expectations of EUSMAT alumni, the EUSMAT had mandated CEval GmbH with the conceptualization and implementation of an alumni survey. The objective was to generate a data base for a further enhancement of the EUSMAT alumni work.
Methodologically, a semi-standardized survey as well as a number of focus group discussions had been designed and conducted among EUSMAT alumni.
Joint departmental strategic evaluation of the AA and BMZ engagement in Iraq
Within the framework of its crisis engagement in Iraq, the Federal Government intends to further embed monitoring and evaluation as instruments of learning and quality assurance, accountability and strategic steering. In this context, a joint evaluation of the Iraq engagement of the Federal Foreign Office (AA) and the Federal Ministry for Economic Cooperation and Development (BMZ) was conducted. This was intended to serve as a pilot project and, if possible, as a model for joint evaluations of both ministries in relevant fields of action and at their interfaces. The main objective of this joint evaluation was joint institutional learning. The AA component of the evaluation was realised by a consortium under the leadership of the GFA Consulting Group, in which CEval GmbH was involved; the BMZ component by the German Evaluation Institute (DEval). Both evaluation teams worked closely together in a so called Arbeitsgemeinschaft.
The aim of the evaluation was to conduct an independent and comprehensive analysis and assessment of the engagement of the two ministries in order to gain and use insights for (joint) steering. The evaluation provided insights into the relevance, effectiveness and overarching impacts, efficiency, sustainability and connectivity as well as coordination, coherence and complementarity of the Iraq engagement of both ministries. Furthermore, the evaluation provided recommendations for optimising the overall engagement and individual measures, structures and processes in order to adjust the current portfolio if necessary. Finally, learning experiences from the AA-BMZ engagement in Iraq were identified for similar engagement in other crisis contexts. The evaluation also served as an accompanying or final performance review of the country portfolio in accordance with the Federal Budget Code (Bundeshaushaltsordnung).
Methodologically, the evaluation was based on a broad mix of qualitative and quantitative data collection and evaluation methods. Due to pandemic-related travel restrictions, the data collection was realised or managed remotely by the consortium in both Germany and Iraq.
Development of results-based framework “Resilience in Strategic Communication”
The digital transformation has also greatly changed the demands on foreign policy and its communication. Whereas it used to be primarily the responsibility of the foreign minister and the global network of foreign missions, foreign policy today has many state and non-state actors. At the same time, global competition for attention has reached a new dimension due to digitalisation as well as numerous new players and alliances. But it has also changed due to a massive shift of communication to social media. Strategic communication today is therefore not only a different way of communicating in other social spaces, but at the same time a confrontation with challenges such as disinformation, propaganda or hate crime and a changed concept of the public, of freedom of expression and its threats.
The aim of the assignment was to provide a decision-making basis for project selection based on clear and predefined criteria on how well a proposed project fits into the overall logic of resilience promotion, as well as to create a basis for evaluating German engagement in the resilience sector. This took the form of developing a results-based framework with associated impact-oriented indicators, which were documented in an accompanying report and embedded in terms of communication theory.
Pilot Project Strategic Evaluation of the Transformation Partnerships with Jordan, Morocco and Tunisia
Together with the GFA Consulting Group, CEval GmbH conducted a strategic evaluation of the transformation partnerships of the Federal Foreign Office (AA), which have been in place since 2011 as part of the German government’s foreign policy to support democratisation processes in Arab countries. The focus of the evaluation was on the areas of rule of law promotion, civil society promotion and foreign cultural and education policy, as well as on the three countries Jordan, Morocco and Tunisia.
The pilot project of the strategic evaluation was in the context of the longer-term goal of expanding the use of strategic evaluation as a steering instrument and anchoring institutional learning in the AA. In addition to examining the procedures and methods actually used to set strategic priorities, the study was therefore also intended to include possibilities for expanding strategic management and improving the link between project implementation and evaluation.
As a starting point for the evaluation, the evaluation team developed results-based framework for the overall strategy of the transformation partnerships, in which both the objectives intended by the AA and possible contextual factors were listed. In addition, results-based frameworks were developed for the priority areas (promotion of the rule of law, promotion of civil society), which were integrated into the impact structure of the overall strategy. Methodologically, both quantitative and qualitative methods were used to collect data in Germany and in the field. The data analysis combined descriptive statistics with qualitative content analysis.
Evaluation and counselling of the project “Digital Civic Education and Counselling”
Against the background of the increasing digitalisation of all areas of society, the Friedrich-Ebert-Stiftung (FES), whose central task is political education and counselling, has launched the internal project “Digital Education and Counselling – FES digital”. Its aim is to anchor digital formats in the FES’s education and advisory services.
The FES had commissioned schmidt evaluation und CEval GmbH to conduct an accompanying evaluation of the project, which was to contribute to clarifying the project goals and indicators as well as those of the individual digital formats and to continuously monitor the achievement of goals.
Methodologically, the evaluation was based on the analysis of existing secondary data, on a benchmarking with regard to suitable target indicators and target values as well as on the implementation of a baseline, interim and final survey by means of online questionnaires and online self-assessments on the respective status of the digitalisation of the political education offers.
Evaluation of the prevention campaign “Mach Dein Passwort stark!”
In January 2019, the Ministry of the Interior of North Rhine-Westphalia (IM NRW) commissioned the NRW State Criminal Police Office (LKA NRW) to develop a prevention and publicity campaign to combat ‘cybercrime’. The aim of the campaign was to strengthen the population in NRW in the safe use of digital end devices and thereby reduce opportunities for cybercrime in the broader sense and make the work of the perpetrators more difficult. The cross-media campaign “Make your password strong!” was launched in October 2020.
CEval GmbH was commissioned to carry out an accompanying evaluation for the campaign with the two sub-areas of process and impact evaluation. The aim of the study was to evaluate the implementation process as well as the effectiveness of the “Make your password strong!” campaign and to derive conclusions and recommendations for future police prevention campaigns based on these results.
The evaluation was based on a broad mix of qualitative and quantitative data collection involving all actors involved in the design and implementation of the campaign as well as the target groups. The design also included an experimental trial set-up for comparative impact evaluation of the different components of the campaign. Descriptive and inferential statistical methods as well as qualitative content analyses were realised within the framework of the data analysis.
Evaluation of the programme “Mentoring Hessen – Frauen in Wissenschaft und Wirtschaft”
Mentoring Hessen is a joint project of Hessian universities and universities of applied sciences to advance the careers of women. The aim is to increase the percentage of women in attractive specialist and management positions in both academia and business. With the help of four different lines of support, young women are supported and guided through the decisive phases of their professional development from student to professor! The close cooperation with renowned companies and research institutions also opens up career opportunities outside of academia. Through mentoring, training and networking, young women are supported in the decisive phases of their professional development.
The goal of the evaluation was to analyse the implementation of the concept and organisational model of Mentoring Hessen in order to develop recommendations for the optimal further development of the programme based on the findings. The evaluation adopted two perspectives of analysis: 1. analysis of the conception, implementation and goal achievement of Mentoring Hessen: Evaluation of the programme conception and implementation, assessment of the achievement of objectives and effectiveness, derivation of recommendations for optimising the conception, implementation and achievement of objectives; 2. Analysis of the interaction of the stakeholders in the joint project Mentoring Hessen: Analysis of the structure and its functionality as well as evaluation of the interaction of the different groups of actors, derivation of recommendations for the optimisation of organisational structure and cooperation.
The data collection combined a document analysis with a comprehensive secondary analysis of existing data from target group surveys, guided interviews and group discussions. The data analysis combined descriptive statistics and qualitative content analysis.
Study program evaluations of the Theaterakademie August Everding
The August Everding Theatre Academy, Munich, is an academic teaching and learning theatre that offers eight courses of study in the field of stage professions at Bachelor’s and Master’s level. These are realised in cooperation with the four Munich universities and the three Munich state theatres.
The Theatre Academy has commissioned CEval GmbH to carry out evaluations of five study programmes. The aim is to collect an evaluation of the theoretical and practical orientation by the students of the various study programmes in the sense of quality development, in order to further optimise the study programmes on the basis of this data. For this purpose, the students are surveyed online using partially standardised questionnaires.
Employee survey on working time and mobile work
The TU Darmstadt (TUD) pursues the goal of developing working time models and regulations for mobile work that are oriented towards the needs of employees. In this way, the satisfaction of employees with the organisation of working time and place of work is to be promoted. To this end, the TUD Executive Board commissioned CEval GmbH to conduct an employee survey on the topics of working hours and mobile work.
The surveys were thematically oriented in such a way that, on the one hand, the lived practice of organising working time and using mobile work was recorded, and on the other hand, the respondents’ satisfaction with this practice as well as suggestions and wishes for the future. Without this being foreseeable when the study was initiated, the Corona pandemic led to considerable adjustments in the design of working conditions during the survey period, especially with regard to mobile work. During the fine-conception of the study, these changes were taken into account by also including questions on the (new) experiences regarding mobile working during the pandemic.
The data collection combined a document analysis with a partially standardised online survey of all employee groups at the TUD as well as guideline-based interviews. The data analysis combined descriptive statistics and qualitative content analysis.
Accompanying evaluation of the support programme “Female Producer Collective” (2022)
The support program “Female Producer Collective” of GOFORMUSIC gGmbH is a special support project for female producers, who receive coaching in the areas of “Creativity and Idea Generation”, “Recording & Sound Design”, “Mixing & Mastering” and “Business & Marketing”. The aim of the project is to specifically promote female producers, to counteract the low quota of women in the production sector and to inspire female musicians to produce. CEval GmbH was commissioned with the accompanying evaluation of the funding project. From a methodological point of view, focus group interviews and a short survey via a standardized questionnaire had been conducted for this purpose.
