Accompanying evaluation of the support programme “Female Producer Collective” (2022)
The support program “Female Producer Collective” of GOFORMUSIC gGmbH is a special support project for female producers, who receive coaching in the areas of “Creativity and Idea Generation”, “Recording & Sound Design”, “Mixing & Mastering” and “Business & Marketing”. The aim of the project is to specifically promote female producers, to counteract the low quota of women in the production sector and to inspire female musicians to produce. CEval GmbH was commissioned with the accompanying evaluation of the funding project. From a methodological point of view, focus group interviews and a short survey via a standardized questionnaire had been conducted for this purpose.
Process Evaluation of the career support program ‘Securing Talents – Shaping the Future’ (2021/2022)
The program ‘Securing Talents – Shaping the Future’ of the Cusanuswerk – Bischöfliche Studienförderung aimed to promote the next generation of female leaders. To this end, the program relies on mentoring elements, i.e. tutorial support for female scholarship holders of German scholarship programs for the gifted by academics with professional experience, training and support measures geared to specific professions, and the establishment of a peer network. Since the start of the career development program in 2007, CEval had conducted external evaluations to ensure the quality of the program. An accompanying evaluation took also place in the eighth program run in 2021/2022. The central task of this evaluation was to provide the Cusanuswerk staff responsible for implementation with timely information relevant to management. In accordance with the program’s objectives, questions regarding satisfaction with the events offered, the quality of the tutorial relationships, and the professional and personal benefits of the relationship for both sides are of particular interest. Methodologically, the evaluation is based on standardized online surveys of mentees and mentors at three points in time.
Process Evaluation of the support program “Collegium Cusanum”
The program ‘Collegium Cusanum’ of the Cusanuswerk – Bischöfliche Studienförderung is aimed at all scholarship holders of the Cusanuswerk (students and doctoral candidates) as well as Altcusaner who aspire to a professional activity in the school or pedagogical field or are already doing so. The support program thus makes a sustainable contribution to the personal development and qualification of teachers and educators. To this end, the program relies on mentoring elements, i.e. tutorial support for Cusanuswerk scholarship holders by teachers and educators with professional experience. The CEval carries out external evaluations accompanying the process for quality assurance purposes. The central task of this evaluation is to provide the Cusanuswerk staff responsible for implementation with timely information relevant to control. Of particular interest, in accordance with the objectives of the program, are questions regarding satisfaction with the events offered, the quality of tutorial relationships, and the professional and personal benefits of the relationship for both parties. Methodologically, the evaluation is based on standardized online surveys of mentees and mentors at three points in time.
Framework contract for the further development and implementation of systematic results-based approaches through monitoring and evaluation for DAAD programs (2020)
The DAAD understands itself as a modern science organization that strives to continuously improve its effectiveness through continuous learning and the willingness to change its work. This approach is formulated in the DAAD’s Integrated Monitoring and Evaluation Concept, which was adopted in 2019. The concept identifies institutional learning, quality assurance, results-based orientation, transparency in processes and legitimacy as important dimensions to which the various monitoring and evaluation activities are geared. Doing so, from an overarching perspective, the strategic orientation of the DAAD and its programs is also to be supported on an evidence-based basis.
The framework contract with CEval GmbH aims to meet the increasing demands for results-based orientation and control in the various funding areas of the DAAD even better by supporting the implementation of results-based orientation and control for existing and new programmes in the AA- and BMBF-funding areas and by accompanying the results-based orientation of newly designed BMZ programmes. To this end, services are provided in the following areas:
Evaluation of the Alexander von Humboldt Foundation’s Research Fellowship Programmes for Researchers from Germany (FLF, JSPS, MOST)
The Alexander von Humboldt Foundation (AvH) enables highly qualified academics from all over the world to spend longer periods of research in Germany and supports the resulting academic and cultural links between Germany and abroad. The Foundation’s network now comprises over 29,000 researchers from all disciplines in more than 140 countries.
To strengthen the Humboldt Network, the AvH’s programme portfolio also includes three research fellowship programmes that support researchers from Germany to work at research institutions abroad. The BMBF-funded Feodor Lynen Research Fellowship Programme (FLF) enables German researchers to spend a longer period of time researching abroad. In this way, the existing Humboldt Network abroad is strengthened and the potential of Humboldt alumni abroad is tapped in their function as academic advisors and “door openers” for (junior) researchers from Germany. For the target countries Japan and Taiwan, the Humboldt Foundation cooperates with the Japan Society for the Promotion of Science (JSPS) and the Ministry of Science and Technology, Taiwan (MOST) within the framework of the JSPS Programme and the MOST Programme. The AvH also offers support services for alumni from all three scholarship programmes from the BMBF funding. These include reintegration assistance, return scholarships, renewed research stays, institute partnerships as well as annual conferences and networking events.
The evaluation is intended to fulfil two functions: On the one hand, it should provide an evidence-based foundation for the further development of the programmes in the sense of a formative evaluation. On the other hand, it is to be used to account to the funding agencies and the public for the implementation and achievement of the goals of the scholarship programmes. The evaluation focuses on the extent to which the Humboldt Network has benefited from the three programmes and whether suitable research fellows have been selected with regard to the programme goals.
The evaluation is characterised by a participatory and multi-method approach. The evaluation combines document and secondary analyses with qualitative (guided intensive interviews) and quantitative survey procedures (partially standardised online surveys).
International consultancy to conduct mapping of Early Childhood Care Development and Education (ECCDE) Centres in Eswatini
The purpose of this assignment was to support the Ministry of Education and Training in Eswatini to conduct a mapping of all early childhood care, development, and education programs (ECCDE). The mapping had been conducted on all centres that provide ECCDE services (formal and informal), including detailing critical information on location, enrollment, infrastructure, and the type of services provided. Results from the ECCDE mapping are expected to inform the development of the Education Sector Strategic Plan 2022 – 2034, including to promote the improvement of access and quality of ECCDE services across the country.
CEval is responsible for the digitalization of the questionnaire with the help of the software Kobocollect, data cleaning and statistical analysis.
Mid-Term Evaluation of the „Palestinian-German Science Bridge” (PGSB)
The program Palestinian-German Science Bridge is a pilot project financed by the German Federal Ministry of Education and Research (BMBF), which is jointly implemented by Forschungszentrum Jülich and the Palestinian Academy for Science and Technology (PALAST). It aims at establishing a joint education and research cooperation between Forschungszentrum Jülich and the Palestinian partner universities. To this end, the program combines measures focusing on student and researcher mobility, scientific exchange among faculty staff, the support of research collaborations and the reintegration of young Palestinian researchers into the Palestinian academic sector.
Forschungszentrum Jülich mandated CEval GmbH to counsel the Forschungszentrum with regard to the conceptualization of a mid-term evaluation of the program and to assist the Forschungszentrum in implementing it. The purpose of the evaluation was to assess the short- and medium-term results of the program as well as the prospects for long-term goal achievement. The objective, further, was to identify potentials for improvement in relation to the program.
Methodologically, the evaluation was based on a mix of qualitative and quantitative data collection and analysis methods. Data collection combines a document and secondary data analysis with guided interviews and semi-standardized online surveys among the project stakeholders. During data analysis, descriptive statistics and qualitative content analysis were applied. Due to the COVID-19 pandemic, the evaluation was planned as a remote evaluation.
Alumni survey of the European School of Materials (EUSMAT)
The European School of Materials (EUSMAT) at Saarland University offers various international study programs at Bachelors, Masters and PhD level. In order to stay in touch with the alumni of these programs, EUSMAT is currently expanding its alumni activities.
To verify how well the current alumni activities fit the needs and to identify expectations of EUSMAT alumni, the EUSMAT had mandated CEval GmbH with the conceptualization and implementation of an alumni survey. The objective was to generate a data base for a further enhancement of the EUSMAT alumni work.
Methodologically, a semi-standardized survey as well as a number of focus group discussions had been designed and conducted among EUSMAT alumni.
Joint departmental strategic evaluation of the AA and BMZ engagement in Iraq
Within the framework of its crisis engagement in Iraq, the Federal Government intends to further embed monitoring and evaluation as instruments of learning and quality assurance, accountability and strategic steering. In this context, a joint evaluation of the Iraq engagement of the Federal Foreign Office (AA) and the Federal Ministry for Economic Cooperation and Development (BMZ) was conducted. This was intended to serve as a pilot project and, if possible, as a model for joint evaluations of both ministries in relevant fields of action and at their interfaces. The main objective of this joint evaluation was joint institutional learning. The AA component of the evaluation was realised by a consortium under the leadership of the GFA Consulting Group, in which CEval GmbH was involved; the BMZ component by the German Evaluation Institute (DEval). Both evaluation teams worked closely together in a so called Arbeitsgemeinschaft.
The aim of the evaluation was to conduct an independent and comprehensive analysis and assessment of the engagement of the two ministries in order to gain and use insights for (joint) steering. The evaluation provided insights into the relevance, effectiveness and overarching impacts, efficiency, sustainability and connectivity as well as coordination, coherence and complementarity of the Iraq engagement of both ministries. Furthermore, the evaluation provided recommendations for optimising the overall engagement and individual measures, structures and processes in order to adjust the current portfolio if necessary. Finally, learning experiences from the AA-BMZ engagement in Iraq were identified for similar engagement in other crisis contexts. The evaluation also served as an accompanying or final performance review of the country portfolio in accordance with the Federal Budget Code (Bundeshaushaltsordnung).
Methodologically, the evaluation was based on a broad mix of qualitative and quantitative data collection and evaluation methods. Due to pandemic-related travel restrictions, the data collection was realised or managed remotely by the consortium in both Germany and Iraq.
Development of results-based framework “Resilience in Strategic Communication”
The digital transformation has also greatly changed the demands on foreign policy and its communication. Whereas it used to be primarily the responsibility of the foreign minister and the global network of foreign missions, foreign policy today has many state and non-state actors. At the same time, global competition for attention has reached a new dimension due to digitalisation as well as numerous new players and alliances. But it has also changed due to a massive shift of communication to social media. Strategic communication today is therefore not only a different way of communicating in other social spaces, but at the same time a confrontation with challenges such as disinformation, propaganda or hate crime and a changed concept of the public, of freedom of expression and its threats.
The aim of the assignment was to provide a decision-making basis for project selection based on clear and predefined criteria on how well a proposed project fits into the overall logic of resilience promotion, as well as to create a basis for evaluating German engagement in the resilience sector. This took the form of developing a results-based framework with associated impact-oriented indicators, which were documented in an accompanying report and embedded in terms of communication theory.
