Process Evaluation of the career support program ‘Securing Talents – Shaping the Future’ (2019/2020)
The program ‘Securing talents – shaping the future’ by the Cusanuswerk aims to support future female leaders. The program focuses on mentoring elements that is tutorial support for scholarship holders by professionally experienced academics. It also focuses on occupationally specific training and support measures as well as on building a peer-network. The CEval has been conducting accompanying external evaluations to ensure quality management since the start of the career support program in 2007. An accompanying evaluation also take place during the seventh program run 2019/2020. The main task of this evaluation was to provide the responsible employees with management-related information. In this context, questions regarding satisfaction with the offered events, quality of tutorial relationships as well as questions regarding professional and personal use of the relationships for both sides are especially relevant. In regards to methods, the evaluation covered standardized questionnaires of mentees and mentors compared at three points in time.
Evaluation of the website www.bne-portal.de
The German Commission for UNESCO is a Quango in the field of multilateral cultural, educational, Research, and media policy. The website www.bne-portal.de is one of their efforts in contributing to the Global Action Programme on Education for Sustainable Development. The website is considered to open the field of education for sustainable development (German: Weltaktionsprogramm Bildung für nachhaltige Entwicklung) to actual as well as future protagonists in Germany. After a re-launch in 2016 and an uptime of one year, CEval GmbH was entrusted to evaluate the website www.bne-portal.de.
The evaluation of the website was focused on (a) clarity, relevance, and appeal of the provided information, (b) attractiveness of design and structure, (c) information accessibility, (d) practical relevance of information, (e) usefulness of provided contact channels, (f) degree of fulfilling user expectations, (g) intensity of use, (h) popularity/attractiveness of its education materials, and (h) popularity/attractiveness of its protagonist database.
The evaluation was based on an online user survey followed by quantitative data analysis.
Evaluation of the Initiative “Schulen – Partner der Zukunft” (PASCH)
The initiative Schools Partners of the future (PASCH), founded in 2008, is part of the central initiatives of the German foreign cultural and educational policy within the current legislature period. PASCH is a worldwide network currently including around 1800 partner schools, which are linked to Germany and especially teach German. The initiative concentrates diversified measures, which are supposed to contribute to establish lively and long-term relations to Germany, and encourage schools, teachers and students to exchange their knowledge and ideas as well as to work together. PASCH is coordinated by the German Federal Foreign Office and is implemented by the Central Office for Schools Abroad, the Goethe Institute, the Pedagogic Exchange Service, the Conference of Education Ministers and the German Academic Exchange Service.
Within the scope of the planned evaluation, PASCH will be analyzed regarding its relevance, efficiency, effectivity, impact and sustainability. It will also be analyzed to what extent the existing offers, processes and structures are suitable instruments for achieving the objectives. Recommendations for possible optimization of the measures, processes and structures and the creation of synergies will be derived on the basis of these results.
The evaluation is divided into two phases: During the first phase, document analyses as well as guided personal and telephone interviews with stakeholders in Germany and selected partner countries (Poland, Turkey and China) will be conducted. The focus of the second phase is then on quantitative online surveys with target groups in China, Turkey and Poland as well as on qualitative and quantitative data collection in another case study country.
Evaluation of the project ‘Media in Libya: Stability through reconciliation’
The object of this evaluation was the project ‘Media in Libya: Stability through reconciliation’ (MLSR) of the DW Akademie. MLSR was a follow-up of the precursor project Media in Libya: Stability through Structure. The overall goal of MLSR was to enable a process of public discourse and eventually reconciliation across a fragmented Libyan society prerequisites for development of democratic reform and constitutional process in the country. By promoting [the] right to access to information to the people of Libya and [the] freedom of expression in Libya (DW Akademie, 2017, p. 4). The approach of MLSR is fourfold (i.e., four activity clusters): (1) supporting Libyan institutions/authorities efforts in the field of media governance, (2) developing a Libyan Media Network, (3) training local freelance journalists from all regions of Libya with a special focus on ethnic minorities and gender, and (4) providing grants for Libyan media projects.
The evaluation of MLSR had been considered as a planned impact evaluation, which needs valid qualitative/quantitative data on the situation at the beginning of MLSR (i.e., baseline). The second stage of evaluation (i.e., interim evaluation) started in January 2019. Comparing the results of baseline and interim evaluation, the evaluators had been able to identify and assess the preliminary effects of MLSR. Moreover, the evaluators may give first recommendations on how to improve MLSR (e.g., identification of [un]successful project strategies, alternative actions). The third stage of evaluation started shortly after the end of MLSR (i.e., January 2021). The main goal of the final evaluation was to assess the effectiveness of MLSR (i.e., achievements in relevant targeted areas).
Methodologically, a robust mixed-method approach was pursued against the background of the very volatile and difficult framework conditions in Libya: Secondary analysis of existing documents/data, content analysis of short written and in-depth oral interviews and statistical analysis of quantitative data from a standardized questionnaire survey.
Evaluation of the International Climate Protection Fellowship for Young climate experts from developing countries
In 2009, the Alexander von Humboldt-Foundation (AvH) initiated the International Climate Protection Fellowship (ICPF) for young climate experts from non-European developing/emerging countries. From 2010 to 2015, they supported 100 future leaders from academic and non-academic fields.
The target Groups of ICPF are young academics and future leaders form industry/politics from developing/emerging countries with higher education. Those Groups participate in developing/applying measures of (1) climate protection, (2) climate-related protection of resources, and (3) adapting to climate Change in developing/emerging countries. The ICPF’s Goal is to accompany their fellows over a long way: 12-months Fellowship for conducting their own research/research-related projects in cooperation with partners/Hosts in Germany, (2) participation in an alumni program (e.g., fundings for further stays in Germany, invitations to their home institutions abroad for their German partnsers/Hosts, invitations to AvH’s Network).
The Evaluation of ICPF examined the extent to which its effects contributed to (1) climate protection and climate-related protection of resources, (2) adapting to climate change in developing/emerging countries. Vital aspects of the evaluation were program-level examination of relevant goal achievements and the analysis of already testable program effects.
The evaluation was based on the DAC criteria. Combining document/secondary analyses with qualitative (extensive partly standarized interviews, case studies)as well as quantitative methods (partly standarized surveys), the evaluation can be characterized by its multi-method approach.
How to manage sustainability and effectiveness?
The actors of German cultural relations and education policy (AKBP) face the challenge that the context of their actions is always in interaction with complex, dynamic political and social contexts. This complicates the creation of knowledge in order to manage projects and thus to secure sustainability and effectiveness of AKBP-projects.
Hence, the aim of this research project was to analyze which methods and tools are suitable in volatile contexts to generate knowledge about effects in changing conditions. In addition, the possibilities to use this knowledge for project management was examined. Therefore, on the one hand a stock taking of existing instruments was made. On the other hand more recent approaches for monitoring and decision-making were analyzed with regard to their applicability in the AKBP.
Evaluation of §§20a and b Police Act of North Rhine-Westphalia (NRW)
Purpose of this project was the scientific support of the evaluation of §§ 20a and b in the North Rhine-Westphalian Police Act which became operative as of 1 of July 2013. It concerned the entitlement of receiving information on telecommunication and media data and included further a norm on data collection by the police with technical means. The new legislation has included the evaluation of impacts and a review of the application in practice with the support of an independent scientific expert after three years of experience. The balance between entitling the police with certain rights to inquire and collect data to protect citizens and at the same time limit the intrusion into private data has been a highly discussed topic in politics and public.
CEval was involved in the concretization and operationalization of leading research questions and in the identification of appropriate indicators. It supported the development of the evaluation design, the data collection instruments as well as data collection itself and data analysis. Data was collected in the form of guideline based interviews, focus group discussions and standardized surveys.
Framework contract for developing and implementing effect-oriented monitoring systems for DAAD programs (2016)
The aim of this project was to empower the DAAD to be able to cope with the increasing challenges in terms of results-based monitoring and control systems. Orienting itself more strongly to actual developmental results and ensuring a stronger focus on its partners’ needs (managing for results), the DAAD, as mediating party, was depending on evidence-based planning, controlling, and reporting of project performing parties (i.e., universities). The monitoring systems that had been implemented in this project was considered to increase the quality of information on program activities – in the context of project performing parties’ accounting as well within interactions between project parties in general (i.e., funding organization, DAAD, and university). Thereby, future monitoring systems promote collaborative learning and strategic development of DAAD’s funding programs.
In this project, CEval was mainly working in the following areas: developing of specific results models and indicator catalogs for existing as well as for future funding programs; adapting existing monitoring processes, procedures, and instruments for project planning, project controlling, and project reporting within DAAD’s funding programs; development of program-specific monitoring concepts and their realization. Moreover, CEval provided training on results-based planning and monitoring (capacity building) for employees of DAAD and of involved universities.
Evaluation of the 2014 Staff/Management Talks at Saarstahl AG
In 2013, Saarstahl AG Management introduced Performance appraisals between Supervisors and their Managers (Mitarbeitergespraeche, short: MAGs) to the company’s human resource Management. These annual talks take place in private, Independent of any particular incident or Event and on a non-hierarchical Basis. The aim of the MAGs at Saarstahl AG is to improve orientation towards the company’s targets and its corporate culture, to promote employee Training and development and to improve cooperation within the Company.
The evaluation of the 2014 MAGs assessed if they were well-received among employees as well as Management in their second year and if Trends concerning acceptance and implementation could be identified.
Special focus has been laid with the following aspects: preparation of employees(seminars, Information material); implementation and communication of target Agreements as well as employee Support and development strategies, General manageability of this tool regarding Content and structure.
Additionally, the study evalated if and how MAGs can be considered as an appropriate tool in human resource Management and if they triggered improvements in Saarstahl AG managerial culture. The aim was to evaluate MAGs in their present form and to identify potentials for optimisation.
In close cooperation with Saarstahl AG, CEval developed and contucted a standardised online Survey among employees at all levels of the Company. Quantitative data of this year’s Evaluation was complemented by data collected in the 2013 MAGs study in order to allow a longitudinal Analysis.
Accompanying evaluation of the project ‘Media in Libya – Stability through Structure"
The „Media in Libya- Stability through Structure“ project had the main goal of supporting the introduction of modern media legislation in Libya and to strengthen the countrys media institutions. In addition, the project should contribute to strengthen independent and professional journalism in the provinces of Libya. The project had a total duration of 30 months and was funded by the European Commission within the SPRING program. The project was controlled and implemented by the Deutsche Welle Academy (Department of Media Development, Team Africa) in cooperation with the Institute for War and Peace Reporting (IWPR). The activities of the project were divided into four activity clusters, of which DW Academy three realised, while IWPR edited one. CEval evaluated in this project clusters 1 to 3, which were in responsibility of the DW Academy.
The evaluation of the project was divided into a baseline, an intermediate and a final evaluation at the end of the implementation phase. CEval followed a mixed-methods approach for data mining. An in-depth analysis of already existing documents was carried out as well as own surveys (standardized online surveys of the participants of the trainings and qualitative guideline interviews with external experts and officials DW Academy).
