Integration of the Programme “Service Agency Communities in One World” (SKEW) in the M&E-System of Engagement Global
Engagement Global gGmbH was established in 2012 as a central institution to promote and coordinate the development policy commitment of civil society, municipal and private sector actors as well as for development policy information and education work. SKEW a program of Engagement Global focuses in particular on the promotion of municipal development cooperation. In order to be able to monitor both the appropriate use of the funds and above all their effectiveness, Engagement Global has created a comprehensive monitoring and evaluation (M&E) system for all departments and programs. Since SKEW was excluded in the development of the monitoring system at that time, CEval and ISPO were commissioned to integrate SKEW into the organization-wide monitoring system.
As with the design of the company-wide M&E system, first the theory of change and indicators were reviewed or developed. Then the objectives and indicators were integrated into the strategical claims of Engagement Global. In order to complete the creation of a monitoring system for the SKEW, a data collection plan were created and the associated data collection instruments had been (further) developed.
Evaluation of the website www.bne-portal.de
The German Commission for UNESCO is a Quango in the field of multilateral cultural, educational, Research, and media policy. The website www.bne-portal.de is one of their efforts in contributing to the Global Action Programme on Education for Sustainable Development. The website is considered to open the field of education for sustainable development (German: Weltaktionsprogramm Bildung für nachhaltige Entwicklung) to actual as well as future protagonists in Germany. After a re-launch in 2016 and an uptime of one year, CEval GmbH was entrusted to evaluate the website www.bne-portal.de.
The evaluation of the website was focused on (a) clarity, relevance, and appeal of the provided information, (b) attractiveness of design and structure, (c) information accessibility, (d) practical relevance of information, (e) usefulness of provided contact channels, (f) degree of fulfilling user expectations, (g) intensity of use, (h) popularity/attractiveness of its education materials, and (h) popularity/attractiveness of its protagonist database.
The evaluation was based on an online user survey followed by quantitative data analysis.
Formative Evaluation of the Out-of-School-Children Initiative (OOSCI)
The focus of this evaluation was the Out-of-School-Children Initiative (OOSCI) by UNICEF and UNESCO Institute for Statistics (UIS). The initiative was launched in 2010 to achieve a reduction in the number of children out of school in participating countries. Means to achieve this goal were to create comprehensive profiles of out-of-School children in each Country, identify the barriers that push children out of school and to propose changes in the policies and strategies of the partner country to address these barriers.
The evaluation aimed at testing the validity of the theory of change and its assumptions. It is used to strengthen the program logic and to provide a formative assessment of the progress towards the overall goal of achieving a reduction in out-of-school children. The evaluation also provides an in-depth insight into benefits for the different stakeholders and their involvement in the initiative. Furthermore, the findings of the evaluation are used to meet the accountability requirements by the donors of OOSCI.
Main source of information of this evaluation have been documents like OOSCI country studies, OOSCI regional studies, government plans and documents from other donors (e.g. the World Bank, UNESCO, foreign donors etc.). In a document review we assessed progress made through OOSCI for a sample of 42 countries and assessed the contents within the document using a 4-step coding. These codings were used for a descriptive analysis and additionally for a Qualitative Comparative Analysis (QCA). QCA enabled us to analyze thoroughly which set of conditions leads to which outcome. Further instrument of data collection has been an online survey to all country offices. Additional qualitative data has been collected in the US and during eight country visits. The evaluation has been conducted together with Proman and further Consultants. CEval took over the lead with regard to the evaluation methodology.
Evaluation of the demographic monitoring tool demowanda: Demand-oriented coaching on self-evaluation
“The BAuA explores the conditions and developments in gainful employment with a view to providing workers with the best possible protection and Support both today and in the future.” (s. BAuA-Website). Leading the project Demografischer Wandel in der Arbeitswelt (demowanda) is one of BAuAs Tasks. demowanda is a demographic monitoring tool, which has been ordered by the Federal Ministry of Labour and Social Affairs.
BAuA realized a self-evaluate of the demowanda project – their collaboration project with BiB, BIBB, DZA, IAB and RKI. The objective of BAuA’s self-evaluation was the analysis and evaluation of the projects implementation process, its outputs, outcomes and impacts.
In order to successfully realize the self-evaluation, BAuA commissioned CEval GmbH to accompany their self-evaluation and to provide them with demand-oriented coaching/training. Employees of BAuA, who are involved in the self-evaluation of demowanda, were enabled to extend their knowledge on self-evaluation. Moreover, coaching/training ensured the conceptual as well as methodological quality of the self-evaluation project.
The demand-oriented coaching included phone and on-site consulting, inhouse trainings and workshops, and support of development evaluation materials/documents.
Process Evaluation of the career support program ‘Securing Talents – Shaping the Future’ (2017-2018)
Process Evaluation of the career support program ‘Securing Talents – Shaping the Future’ (2017/2018)
The program ‘Securing talents – shaping the future’ by the Cusanuswerk aims to support future female leaders. The program focuses on mentoring elements that is tutorial support for scholarship holders by professionally experienced academics. It also focuses on occupationally specific training and support measures as well as on building a peer-network. The CEval has been conducting accompanying external evaluations to ensure quality management since the start of the career support program in 2007. An accompanying evaluation took also place during the sixth program run 2017/18. The main task of this evaluation was to provide the responsible employees with management-related information. In this context, questions regarding satisfaction with the offered events, quality of tutorial relationships as well as questions regarding professional and personal use of the relationships for both sides were especially relevant. In regards to methods, the evaluation covers standardized questionnaires of mentees and mentors compared at three points in time.
External Evaluation of the Netherlands Fellowship Programme (NFP II) and Netherlands Initiative for Capacity Development in Higher Education (NICHE II)
On behalf of the Dutch Ministry of Foreign Affairs (MFA) the CEval implemented an external Evaluation of the Dutch Fellowship Programme (NFP II) and Netherlands Initiative for Capacity Development in Higher Education (NICHE II). Financed by the MFA and administered by the Netherlands Organisation for International Cooperation in Higher Education ‘EP-Nuffic’, these programs offer capacity development in higher education for developing partner countries.
The overall aim of NFP and NICHE is to help reduce quantitative and qualitative shortage of trained professional staff in developing countries and to build sustainable capacity within the poverty reduction Framework. The NFP provides Fellowships to mid-career professionals to participate in Trainings and postgraduate courses in the Netherlands. NICHE is a capacity building programme to improve the quality, relevance and gender sensitiveness of post-secondary education in NICHE-countries.
The evaluation covered the period of 2002-2015 while a focus was on the latest project phase since 2012 which put an additional emphasis on economic diplomacy and closer alignment with the four priority themes of the Dutch development cooperation strategy (water, sexual reproductive health and rights, security and rule of law and food security). The purpose of the evaluation was to account for the Netherlands funding provided for the two programs under the current project cycle and to gain lessons for future policy development and implementation. Within the scope of the evaluation, both programs were analysed regarding the OECD/DAC criteria of relevance, efficiency, effectiveness, impact and sustainability. Recommendations for possible optimization of the program were developed on the basis of the findings. The evaluation was conducted in cooperation with ACE Europe, a Belgian consultancy.
Process Evaluation of the career support program ‘Securing Talents – Shaping the Future’ (2019/2020)
The program ‘Securing talents – shaping the future’ by the Cusanuswerk aims to support future female leaders. The program focuses on mentoring elements that is tutorial support for scholarship holders by professionally experienced academics. It also focuses on occupationally specific training and support measures as well as on building a peer-network. The CEval has been conducting accompanying external evaluations to ensure quality management since the start of the career support program in 2007. An accompanying evaluation also take place during the seventh program run 2019/2020. The main task of this evaluation was to provide the responsible employees with management-related information. In this context, questions regarding satisfaction with the offered events, quality of tutorial relationships as well as questions regarding professional and personal use of the relationships for both sides are especially relevant. In regards to methods, the evaluation covered standardized questionnaires of mentees and mentors compared at three points in time.
Evaluation of the Initiative “Schulen – Partner der Zukunft” (PASCH)
The initiative Schools Partners of the future (PASCH), founded in 2008, is part of the central initiatives of the German foreign cultural and educational policy within the current legislature period. PASCH is a worldwide network currently including around 1800 partner schools, which are linked to Germany and especially teach German. The initiative concentrates diversified measures, which are supposed to contribute to establish lively and long-term relations to Germany, and encourage schools, teachers and students to exchange their knowledge and ideas as well as to work together. PASCH is coordinated by the German Federal Foreign Office and is implemented by the Central Office for Schools Abroad, the Goethe Institute, the Pedagogic Exchange Service, the Conference of Education Ministers and the German Academic Exchange Service.
Within the scope of the planned evaluation, PASCH will be analyzed regarding its relevance, efficiency, effectivity, impact and sustainability. It will also be analyzed to what extent the existing offers, processes and structures are suitable instruments for achieving the objectives. Recommendations for possible optimization of the measures, processes and structures and the creation of synergies will be derived on the basis of these results.
The evaluation is divided into two phases: During the first phase, document analyses as well as guided personal and telephone interviews with stakeholders in Germany and selected partner countries (Poland, Turkey and China) will be conducted. The focus of the second phase is then on quantitative online surveys with target groups in China, Turkey and Poland as well as on qualitative and quantitative data collection in another case study country.
Evaluation of the project ‘Media in Libya: Stability through reconciliation’
The object of this evaluation was the project ‘Media in Libya: Stability through reconciliation’ (MLSR) of the DW Akademie. MLSR was a follow-up of the precursor project Media in Libya: Stability through Structure. The overall goal of MLSR was to enable a process of public discourse and eventually reconciliation across a fragmented Libyan society prerequisites for development of democratic reform and constitutional process in the country. By promoting [the] right to access to information to the people of Libya and [the] freedom of expression in Libya (DW Akademie, 2017, p. 4). The approach of MLSR is fourfold (i.e., four activity clusters): (1) supporting Libyan institutions/authorities efforts in the field of media governance, (2) developing a Libyan Media Network, (3) training local freelance journalists from all regions of Libya with a special focus on ethnic minorities and gender, and (4) providing grants for Libyan media projects.
The evaluation of MLSR had been considered as a planned impact evaluation, which needs valid qualitative/quantitative data on the situation at the beginning of MLSR (i.e., baseline). The second stage of evaluation (i.e., interim evaluation) started in January 2019. Comparing the results of baseline and interim evaluation, the evaluators had been able to identify and assess the preliminary effects of MLSR. Moreover, the evaluators may give first recommendations on how to improve MLSR (e.g., identification of [un]successful project strategies, alternative actions). The third stage of evaluation started shortly after the end of MLSR (i.e., January 2021). The main goal of the final evaluation was to assess the effectiveness of MLSR (i.e., achievements in relevant targeted areas).
Methodologically, a robust mixed-method approach was pursued against the background of the very volatile and difficult framework conditions in Libya: Secondary analysis of existing documents/data, content analysis of short written and in-depth oral interviews and statistical analysis of quantitative data from a standardized questionnaire survey.
Evaluation of the International Climate Protection Fellowship for Young climate experts from developing countries
In 2009, the Alexander von Humboldt-Foundation (AvH) initiated the International Climate Protection Fellowship (ICPF) for young climate experts from non-European developing/emerging countries. From 2010 to 2015, they supported 100 future leaders from academic and non-academic fields.
The target Groups of ICPF are young academics and future leaders form industry/politics from developing/emerging countries with higher education. Those Groups participate in developing/applying measures of (1) climate protection, (2) climate-related protection of resources, and (3) adapting to climate Change in developing/emerging countries. The ICPF’s Goal is to accompany their fellows over a long way: 12-months Fellowship for conducting their own research/research-related projects in cooperation with partners/Hosts in Germany, (2) participation in an alumni program (e.g., fundings for further stays in Germany, invitations to their home institutions abroad for their German partnsers/Hosts, invitations to AvH’s Network).
The Evaluation of ICPF examined the extent to which its effects contributed to (1) climate protection and climate-related protection of resources, (2) adapting to climate change in developing/emerging countries. Vital aspects of the evaluation were program-level examination of relevant goal achievements and the analysis of already testable program effects.
The evaluation was based on the DAC criteria. Combining document/secondary analyses with qualitative (extensive partly standarized interviews, case studies)as well as quantitative methods (partly standarized surveys), the evaluation can be characterized by its multi-method approach.
