Evaluation of the German-Argentinean University Center (DAHZ-CUAA)
The German-Argentinean University Center (DAHZ-CUAA) is a binational university center that aims at promoting and strengthening the institutional, academic, and scientific cooperation between Argentina and Germany. The main objective is the creation of innovative, binational study programs with double degrees. In order to achieve this objective, there are three program lines to support universities in both countries, i.e., the German-Argentinean program to promote binational study programs with double degrees (main program line), the I.DEAR program (Engineers Germany-Argentina), and the program to promote accompanying research projects.
To gain insights into the successes, potentials for improvement, and the progress made in the three program lines and to be able to further develop the programs and to make them even more effective on that basis, the German office of the center mandated CEval GmbH in cooperation with smep consult e.U. with an evaluation of the center’s work. The mission of the evaluation was to assess the relevance, effectiveness (i.e., goal achievement and impact), efficiency, and sustainability of the center and its program lines and to put forth recommendations that allow further development and possibly improving the program lines in the future.
Methodologically, the evaluation was based on a combination of qualitative and quantitative methods of data collection and analysis. It included the analysis of documents and secondary data, site visits in Germany and Argentina, guided interviews, group discussions and online surveys.
Evaluation of the ASG Mentoring Programme 2022/2023
The overriding goal of the non-profit Agrarsoziale Gesellschaft e.V. is to work for the improvement of living conditions in rural areas and in agriculture. Qualified and committed people are a central resource for the positive development of rural areas. Against this backdrop, an ASG mentoring programme was created in which students and graduates who would like to become professionally involved in and for rural areas are supported on their career path over a period of 18 months. The ASG Mentoring Programme is based on the three classic mentoring components: The tutorial support of mentees through the formation of mentoring tandems with leaders from science, administration, politics, non-governmental organisations and business, the acquisition of competencies by the mentees within the framework of the accompanying seminars of the programme, ASG conferences/seminars and the development of personal contacts and networks through the professional exchange with all stakeholder groups within the programme as well as in the context of the ASG conferences and seminars. CEval GmbH was commissioned with the accompanying evaluation of the mentoring programme. The central task of this evaluation is to provide the ASG staff who are responsible for the implementation of the project with timely and relevant information. Of particular interest are questions about satisfaction with the seminars offered, the quality of the tutorial relationships, and the professional and personal benefits of the relationship for both sides. Methodologically, the evaluation is based on standardised online surveys of the mentees and/or mentors at six points in time and on qualitative interviews of the programme leaders and mentors as well as on (focus) group discussions with the mentees.
Implementation of an evaluation system for courses at the August Everding Theatre Academy
The August Everding Theatre Academy offers various artistic study programmes at Bachelor’s and Master’s level. In order to determine the students’ satisfaction with individual courses and to identify potential for improvement, the August Everding Theatre Academy has commissioned CEval GmbH to implement an evaluation system for courses at the August Everding Theatre Academy. The aim is to create a regular database for the further development and optimisation of the courses. Methodologically, semi-standardised surveys will be designed for various teaching formats, which can be used for the evaluations of different courses (practically oriented and theoretically oriented courses as well as individual lessons).
Synthesis and Meta-Evaluation of Project and Program Evaluations of the Centers for German and European Studies (CGES)
In view of its long-term goal to contribute to the internationalization of the science and innovation systems in Germany, Europe and worldwide and to shaping education and research in relation to Germany and Europe, the DAAD supports interdisciplinary Centers for German and European Studies (CGES) at universities abroad. The Centers train future experts in the field of German and European Studies, among others, by offering respective study programs. They also engage in a discourse on topics related to Germany and Europe with stakeholders from science, politics, and the interested public.
To continuously improve the effectiveness of its work, the DAAD had mandated CEval GmbH with the implementation of a meta-evaluation and synthesis regarding the CGES. The object of the study were the project and program evaluations on the CGES implemented between 2015 and 2020. The task entailed a criteria-based synthesis of the findings documented in the evaluation reports as well as an assessment of the quality of the evaluations themselves (meta-evaluation). The objective of the study was, among others, to assess the effectiveness and goal achievement in relation to the CGES from a birds-eye-view, to identify potentials for improvement and to develop recommendations for the future support of the CGES as well as in view of any future CGES evaluations.
Methodologically, the study was desk-based. For the meta-evaluation, the evaluation team developed an assessment scheme as the primary data collection tool. On that basis, the evaluation reports were systematically analyzed and assessed according to a set of previously defined quality criteria and in relation to six different dimensions of analysis. Similarly, an analysis grid was developed for the purpose of the synthesis. It served to derive a coding scheme and, on that basis, implement a qualitative content analysis of the evaluation reports, which then allowed for a synthesize of the evaluation findings.
Mid-Term Evaluation of the „Palestinian-German Science Bridge” (PGSB)
The program Palestinian-German Science Bridge is a pilot project financed by the German Federal Ministry of Education and Research (BMBF), which is jointly implemented by Forschungszentrum Jülich and the Palestinian Academy for Science and Technology (PALAST). It aims at establishing a joint education and research cooperation between Forschungszentrum Jülich and the Palestinian partner universities. To this end, the program combines measures focusing on student and researcher mobility, scientific exchange among faculty staff, the support of research collaborations and the reintegration of young Palestinian researchers into the Palestinian academic sector.
Forschungszentrum Jülich mandated CEval GmbH to counsel the Forschungszentrum with regard to the conceptualization of a mid-term evaluation of the program and to assist the Forschungszentrum in implementing it. The purpose of the evaluation was to assess the short- and medium-term results of the program as well as the prospects for long-term goal achievement. The objective, further, was to identify potentials for improvement in relation to the program.
Methodologically, the evaluation was based on a mix of qualitative and quantitative data collection and analysis methods. Data collection combines a document and secondary data analysis with guided interviews and semi-standardized online surveys among the project stakeholders. During data analysis, descriptive statistics and qualitative content analysis were applied. Due to the COVID-19 pandemic, the evaluation was planned as a remote evaluation.
Alumni survey of the European School of Materials (EUSMAT)
The European School of Materials (EUSMAT) at Saarland University offers various international study programs at Bachelors, Masters and PhD level. In order to stay in touch with the alumni of these programs, EUSMAT is currently expanding its alumni activities.
To verify how well the current alumni activities fit the needs and to identify expectations of EUSMAT alumni, the EUSMAT had mandated CEval GmbH with the conceptualization and implementation of an alumni survey. The objective was to generate a data base for a further enhancement of the EUSMAT alumni work.
Methodologically, a semi-standardized survey as well as a number of focus group discussions had been designed and conducted among EUSMAT alumni.
Joint departmental strategic evaluation of the AA and BMZ engagement in Iraq
Within the framework of its crisis engagement in Iraq, the Federal Government intends to further embed monitoring and evaluation as instruments of learning and quality assurance, accountability and strategic steering. In this context, a joint evaluation of the Iraq engagement of the Federal Foreign Office (AA) and the Federal Ministry for Economic Cooperation and Development (BMZ) was conducted. This was intended to serve as a pilot project and, if possible, as a model for joint evaluations of both ministries in relevant fields of action and at their interfaces. The main objective of this joint evaluation was joint institutional learning. The AA component of the evaluation was realised by a consortium under the leadership of the GFA Consulting Group, in which CEval GmbH was involved; the BMZ component by the German Evaluation Institute (DEval). Both evaluation teams worked closely together in a so called Arbeitsgemeinschaft.
The aim of the evaluation was to conduct an independent and comprehensive analysis and assessment of the engagement of the two ministries in order to gain and use insights for (joint) steering. The evaluation provided insights into the relevance, effectiveness and overarching impacts, efficiency, sustainability and connectivity as well as coordination, coherence and complementarity of the Iraq engagement of both ministries. Furthermore, the evaluation provided recommendations for optimising the overall engagement and individual measures, structures and processes in order to adjust the current portfolio if necessary. Finally, learning experiences from the AA-BMZ engagement in Iraq were identified for similar engagement in other crisis contexts. The evaluation also served as an accompanying or final performance review of the country portfolio in accordance with the Federal Budget Code (Bundeshaushaltsordnung).
Methodologically, the evaluation was based on a broad mix of qualitative and quantitative data collection and evaluation methods. Due to pandemic-related travel restrictions, the data collection was realised or managed remotely by the consortium in both Germany and Iraq.
Development of results-based framework “Resilience in Strategic Communication”
The digital transformation has also greatly changed the demands on foreign policy and its communication. Whereas it used to be primarily the responsibility of the foreign minister and the global network of foreign missions, foreign policy today has many state and non-state actors. At the same time, global competition for attention has reached a new dimension due to digitalisation as well as numerous new players and alliances. But it has also changed due to a massive shift of communication to social media. Strategic communication today is therefore not only a different way of communicating in other social spaces, but at the same time a confrontation with challenges such as disinformation, propaganda or hate crime and a changed concept of the public, of freedom of expression and its threats.
The aim of the assignment was to provide a decision-making basis for project selection based on clear and predefined criteria on how well a proposed project fits into the overall logic of resilience promotion, as well as to create a basis for evaluating German engagement in the resilience sector. This took the form of developing a results-based framework with associated impact-oriented indicators, which were documented in an accompanying report and embedded in terms of communication theory.
Pilot Project Strategic Evaluation of the Transformation Partnerships with Jordan, Morocco and Tunisia
Together with the GFA Consulting Group, CEval GmbH conducted a strategic evaluation of the transformation partnerships of the Federal Foreign Office (AA), which have been in place since 2011 as part of the German government’s foreign policy to support democratisation processes in Arab countries. The focus of the evaluation was on the areas of rule of law promotion, civil society promotion and foreign cultural and education policy, as well as on the three countries Jordan, Morocco and Tunisia.
The pilot project of the strategic evaluation was in the context of the longer-term goal of expanding the use of strategic evaluation as a steering instrument and anchoring institutional learning in the AA. In addition to examining the procedures and methods actually used to set strategic priorities, the study was therefore also intended to include possibilities for expanding strategic management and improving the link between project implementation and evaluation.
As a starting point for the evaluation, the evaluation team developed results-based framework for the overall strategy of the transformation partnerships, in which both the objectives intended by the AA and possible contextual factors were listed. In addition, results-based frameworks were developed for the priority areas (promotion of the rule of law, promotion of civil society), which were integrated into the impact structure of the overall strategy. Methodologically, both quantitative and qualitative methods were used to collect data in Germany and in the field. The data analysis combined descriptive statistics with qualitative content analysis.
Evaluation and counselling of the project “Digital Civic Education and Counselling”
Against the background of the increasing digitalisation of all areas of society, the Friedrich-Ebert-Stiftung (FES), whose central task is political education and counselling, has launched the internal project “Digital Education and Counselling – FES digital”. Its aim is to anchor digital formats in the FES’s education and advisory services.
The FES had commissioned schmidt evaluation und CEval GmbH to conduct an accompanying evaluation of the project, which was to contribute to clarifying the project goals and indicators as well as those of the individual digital formats and to continuously monitor the achievement of goals.
Methodologically, the evaluation was based on the analysis of existing secondary data, on a benchmarking with regard to suitable target indicators and target values as well as on the implementation of a baseline, interim and final survey by means of online questionnaires and online self-assessments on the respective status of the digitalisation of the political education offers.
