Joint departmental strategic evaluation of the Federal Government’s civilian engagement in Afghanistan
The subject of the interministerial strategic evaluation was the Federal Government’s civilian engagement in Afghanistan in the years from 2013 until the nearly complete withdrawal of international organisations from Afghanistan in August 2021. The evaluation focused on the activities of the Federal Foreign Office (AA), the Federal Ministry for Economic Cooperation and Development (BMZ) and the Federal Ministry of the Interior and Community (BMI).
The evaluation was carried out by a working group of three evaluation teams: the GFA Consulting Group consortium for the Federal Foreign Office, the evaluation team of the German Institute for Development Evaluation (DEval) for the Federal Ministry for Economic Cooperation and Development (BMZ) and the evaluation team of the German Police University (DHPol) for the Federal Ministry of the Interior.
The evaluation focused on the areas of stabilisation, police development, development cooperation and reconstruction, humanitarian aid and foreign cultural and educational policy (AKBP) and thus on the measures implemented within these areas by the ministries in accordance with their respective departmental responsibilities. The main objective of the strategic evaluation was to conduct an independent and comprehensive analysis and assessment of the German government’s civilian engagement in Afghanistan in the period 2013-2021 in order to generate insights that can be used for the management of future civil engagement by the German government in particularly fragile contexts characterised by massive violent conflicts.
It was used for joint institutional learning, in particular by the AA, BMI and BMZ, as well as for purposes of political accountability and performance review.
The evaluation was structured along the six OECD/DAC criteria. Three levels of analysis were distinguished: The political (decision-making) level, the strategic level of the ministerial managers and the operational level of the implementing organisations. The evaluation followed a theory-based approach: Theories of Change (ToC) were developed for the political level of the federal government and for the strategic levels of the three ministries. The study, which was designed as a remote evaluation due to the security situation in Afghanistan, was methodologically based on a mix of qualitative and quantitative data collection and analysis methods. It combined document and secondary data analyses with guided individual and group interviews.
Tracer study of alumni of the Max Weber Foundation’s institutes abroad 2005-2022
The Max Weber Foundation (MWS) is one of the leading organisations supporting German research in humanities and social sciences abroad. Therefore, the foundation maintains 11 institutes located around the world, which have a bridging function between the host countries and Germany. The institutes abroad play a particularly important role in the professional development and support of young academics in an international and interdisciplinary environment.
CEval GmbH has been contracted by the MWS to carry out a tracer study. The study will analyse the professional careers of former employees who worked at one of the MWS’s institutes abroad in a qualification position or a long-term scholarship and who left between 2005 and 2022. The aim of the study is to evaluate the success and thus the competitiveness of the promotion of young academics at the institutes.
The tracer study follows a mixed-methods approach combining qualitative and quantitative instruments for data collection and analysis. An online survey, semi-structured interviews and a secondary data analysis will be applied.
Evaluation of the EU Home Affairs Funds: Mid-term evaluations of the Internal Security Fund (ISF) and the Instrument for Financial Support for Border Management and Visa Policy (BMVI)
Under the responsibility of the Directorate-General for Migration and Home Affairs (DG HOME), the European Commission (EU COM) provides the Internal Security Fund (ISF; managed by the Federal Criminal Police Office (BKA)), the Border Management and Visa Instrument (BMVI, managed by the Federal Police) and the Asylum, Migration and Integration Fund (AMIF, managed by the Federal Office for Migration and Refugees (BAMF)). Together, these form the so-called “Home Affairs Funds”.
The member states are obliged to prepare mid-term and final evaluations for the EU Commission on the implementation of the measures financed by the individual funds and the degree to which the objectives of the national programmes supported by the Home Affairs Funds have been achieved. The main users of the evaluations are the EU (DG Home), the fund-managing authorities BKA, BPOL and BAMF and their monitoring committees.
The aim of the evaluations is to assess the results and outcomes of the programmes and measures co-financed by the Home Affairs Fund based on the criteria of effectiveness, efficiency, relevance, coherence and EU added value. The coherence between the individual funds is of particular interest. In addition to assessing success, the evaluations should also develop recommendations for improving the quality of the design and implementation of the national programmes and analyse the coherence and use of synergy potential between the individual funds.
After the Gesellschaft für Internationale Zusammenarbeit (GIZ) carried out the mid-term evaluation for the ISF on behalf of the BKA in 2017, it has now been commissioned again to carry out the final evaluation of the ISF for the 2014-2020 funding period and the mid-term evaluation of the ISF, the BMVI and the AMIF for the 2021-2027 funding period.
In the course of this, GIZ commissioned CEval GmbH to carry out the
> Final evaluation of the ISF, security sub-sector and border sub-sector, 2014-2021 funding period,
> Mid-term evaluation of the ISF, current funding period 2021-2027 and
> Mid-term evaluation of the BMVI, current funding period 2021-2027
to be carried out.
Various survey procedures are used to triangulate sources and methods:
> Research of all relevant programme and project documents as well as monitoring data,
> Semi-standardised online survey with project participants and their implementation partners as well as other potentially eligible applicants and
> Guided qualitative interviews with programme and project managers.
Evaluation of the admission procedures to scholarship programs of the Hans Böckler Foundation (HBS)
The Hans Böckler Foundation (HBS) awards scholarships to secondary school leavers, artists and students as well as to doctoral candidates in all academic fields. To obtain a scholarship, applicants go through a multi-stage and meanwhile digitalized admission procedure in which their appropriateness is assessed on the basis of a series of criteria. CEval GmbH was commissioned by the HBS to evaluate the application procedures. In particular, the CEval GmbH examines the appropriateness, purposefulness and efficiency of the procedures as well as the fairness and comprehensibility of the test criteria.
Therefore, the CEval team orientates itself on the OECD-DAC criteria (relevance, coherence, efficiency, effectiveness, impact, sustainability), which are used to systematically develop an analytical matrix. The evaluation follows a mixed-methods approach combining qualitative and quantitative instruments for data collection and analysis. Semi-structured guided interviews, qualitative short interviews, an online survey and a secondary data analysis will be applied.
Evaluation “RIGHT TO EDUCATION, RIGHT TO LEARN” Finland’s Development Cooperation in the Education Sector
The Unit for Development Evaluation of the Ministry for Foreign Affairs (MFA) of Finland had commissioned a strategic evaluation to inform the Ministry on how the various development policy and cooperation actors have succeeded in stepping up Finland’s global efforts in the education sector. The main objectives of the evaluation were three-fold: First, to analyse the relevance, coherence and efficiency of the response to the global learning crisis and the quality of education by the various development cooperation actors involved. Second, to analyse the effectiveness and results achieved in the area of development cooperation in stepping up Finland’s global role in education sector development in developing countries in line with its development policy objectives, and the value of the multi-actor approach therein. Third, to explore and document alternative future approaches for 2023-2030 in which Finland’s global role and response to the learning crisis and quality education could be the most relevant, coherent, efficient, and effective.
The evaluation’s overarching analysis framework was theory-based contribution analysis, based on a nested (macro-micro level) theory of change. The evaluation team used a mix of data collection and analysis methods in order to answer the evaluation questions. The methods comprised of desk review; key informant interviews; three Country Case Studies (Ethiopia, Nepal and Palestine); two desk-based Thematic studies (on basic education teacher training in Mozambique, and TVET and life skills training in Ukraine); and especially focusing on the future, the Delphi method of forecasting was used (on-line survey and interviews).
Perspective study on the DAAD program “HAW.International” (2023/2024)
The program ‘Securing Talents – Shaping the Future’ of the Cusanuswerk – Bischöfliche Studienförderung aims to promote the next generation of female leaders. To this end, the program relies on mentoring elements, i.e. tutorial support for female scholarship holders of German scholarship programs for the gifted by academics with professional experience, training and support measures geared to specific professions, and the establishment of a peer network. Since the start of the career development program in 2007, CEval has conducted external evaluations to ensure the quality of the program. An accompanying evaluation will also take place in the program run in 2023/2024. The central task of this evaluation is to provide the Cusanuswerk staff responsible for implementation with timely information relevant to management. In accordance with the program’s objectives, questions regarding satisfaction with the events offered, the quality of the tutorial relationships, and the professional and personal benefits of the relationship for both sides are of particular interest. Methodologically, the evaluation is based on standardized baseline and final online surveys of mentees and mentors.
Perspective study on the DAAD program “HAW.International”
The German Academic Exchange Service (DAAD) is the world’s largest funding organization for the international exchange of students and academics. In addition to awarding scholarships, the DAAD also promotes the internationalization of German universities. The DAAD funding program ‘HAW.International’ has been offered since 2019 and is aimed exclusively at universities of applied sciences (HAW). As part of the program, students and lecturers at HAW are supported with individual scholarships as well as structure-building and model and cooperation projects. In the long term, the program is intended to bring about a comprehensive and sustainable internationalization of HAW and promote its strategic positioning as an innovative university model for practice-oriented academic education worldwide.
In preparation for a further funding phase from 2025, the perspective study summarised the results of the funding to date and examined the need for further support at HAW to strengthen their internationality and international visibility.
The study was designed as a monitoring-based secondary data study and therefore based its analysis primarily on the qualitative and quantitative monitoring data on the program available at the DAAD. In addition, qualitative individual and group interviews were also utilized. The results should provide the DAAD with a basis for planning and designing the second program generation of HAW.International.
Framework contract for the further development and implementation of systematic results-based approaches through monitoring and evaluation for DAAD programs (2023)
The DAAD understands itself as a modern science organization that strives to continuously improve its effectiveness through continuous learning and the willingness to change its work. This approach is formulated in the DAAD’s Integrated Monitoring and Evaluation Concept, which was adopted in 2019. The concept identifies institutional learning, quality assurance, results-based orientation, transparency in processes and legitimacy as important dimensions to which the various monitoring and evaluation activities are geared. Doing so, from an overarching perspective, the strategic orientation of the DAAD and its programs is also to be supported on an evidence-based basis.
The framework agreement focuses on impact-oriented program planning, impact-oriented monitoring and target achievement and thus on further development and learning processes as well as on reporting obligations within the framework of performance monitoring. However, it also focuses on the effective and efficient use of the data generated by the Results-oriented Monitoring (RoM) in the context of monitoring-based analyses and evaluations and can thus make a significant contribution to strengthening and increasing the benefits of the RoM.
For this purpose, services are provided in the following areas:
– Development of results frameworks and indicator catalogues
– Development of program-specific monitoring instruments and revision of the program documents according to the requirements of the monitoring concept
– Data processing and analysis
– Monitoring based analyses and evaluations
– Internal and external workshops for the implementation of RoM
– Support in the context of monitoring visits to project partners
– Consultations on further approaches and methods in the field of monitoring and evaluation
– Further consulting and training services
Support for the evaluation of the Azubi Kolleg Lübeck
The Lübeck Trainee Kolleg is a programme of the Joachim Herz Foundation and the Lübeck Possehl Foundation, which supports 25 Lübeck trainees over two years in their personal development and promotes social skills. Seminars, workshops and training sessions are designed to enable the trainees to recognise and develop their strengths and potential and to increase their self-confidence, sense of responsibility and social commitment. In the block events, further training is offered on three major thematic blocks: a) communication, b) society/ politics/ economy and c) cultural education. Further goals of the programme are the exchange between the trainees, the joint learning process and the development of a network. In the long term, the programme is intended to contribute to the personal development and professional establishment of the trainees and to increase the attractiveness of vocational training. In addition, it is intended to give the development of talented trainees a greater presence in society. As a pilot project, the Azubi Kolleg is designed to run for five years.
In 2023, CEval GmbH was assigned again to support the evaluation of the programme. On the basis of the results-based framework and indicator catalogue developed in the previous project, the baseline data of the second cohort had been analyzed in 2023 in comparison with the data of the first cohort. In addition, online surveys among trainees, teachers and supervisors as well as focus group interviews with trainees and guided interviews with teachers and supervisors at the end of the programme are planned as the basis for a before-and-after comparison had been conducted.
Ex-ante survey in the programme “Azubi Kolleg Lübeck”
The Lübeck Trainee Kolleg is a programme of the Joachim Herz Foundation and the Lübeck Possehl Foundation, which supports 25 Lübeck trainees over two years in their personal development and promotes social skills. Seminars, workshops and training sessions are designed to enable the trainees to recognise and develop their strengths and potential and to increase their self-confidence, sense of responsibility and social commitment. In the block events, further training is offered on three major thematic blocks: a) communication, b) society/ politics/ economy and c) cultural education. Further goals of the programme are the exchange between the trainees, the joint learning process and the development of a network. In the long term, the programme is intended to contribute to the personal development and professional establishment of the trainees and to increase the attractiveness of vocational training. In addition, it is intended to give the development of talented trainees a greater presence in society. As a pilot project, the Azubi Kolleg is designed to run for five years.
In order to continuously improve the programme, CEval GmbH was commissioned to strengthen the impact orientation of the programme, therefore, in a first step, CEval GmbH developed a results-based framework and an indicator catalogue. On this basis, the next step was to conduct ex-ante surveys in the form of online surveys among the trainees and among the persons who supervise the trainees in the company (e.g. as a trainer, management, personnel department, etc.) or at school (e.g. as a class teacher, head of department, etc.). On the other hand, the surveys were intended to provide information about the trainees’ self-assessment and others’ assessment of their skills, personality traits, leisure activities or fields of interest at the beginning of their participation in the programme.
