Evaluation of the EU Home Affairs Funds: Mid-term evaluations of the Internal Security Fund (ISF) and the Instrument for Financial Support for Border Management and Visa Policy (BMVI)
Under the responsibility of the Directorate-General for Migration and Home Affairs (DG HOME), the European Commission (EU COM) provides the Internal Security Fund (ISF; managed by the Federal Criminal Police Office (BKA)), the Border Management and Visa Instrument (BMVI, managed by the Federal Police) and the Asylum, Migration and Integration Fund (AMIF, managed by the Federal Office for Migration and Refugees (BAMF)). Together, these form the so-called “Home Affairs Funds”.
The member states are obliged to prepare mid-term and final evaluations for the EU Commission on the implementation of the measures financed by the individual funds and the degree to which the objectives of the national programmes supported by the Home Affairs Funds have been achieved. The main users of the evaluations are the EU (DG Home), the fund-managing authorities BKA, BPOL and BAMF and their monitoring committees.
The aim of the evaluations is to assess the results and outcomes of the programmes and measures co-financed by the Home Affairs Fund based on the criteria of effectiveness, efficiency, relevance, coherence and EU added value. The coherence between the individual funds is of particular interest. In addition to assessing success, the evaluations should also develop recommendations for improving the quality of the design and implementation of the national programmes and analyse the coherence and use of synergy potential between the individual funds.
After the Gesellschaft für Internationale Zusammenarbeit (GIZ) carried out the mid-term evaluation for the ISF on behalf of the BKA in 2017, it has now been commissioned again to carry out the final evaluation of the ISF for the 2014-2020 funding period and the mid-term evaluation of the ISF, the BMVI and the AMIF for the 2021-2027 funding period.
In the course of this, GIZ commissioned CEval GmbH to carry out the
> Final evaluation of the ISF, security sub-sector and border sub-sector, 2014-2021 funding period,
> Mid-term evaluation of the ISF, current funding period 2021-2027 and
> Mid-term evaluation of the BMVI, current funding period 2021-2027
to be carried out.
Various survey procedures are used to triangulate sources and methods:
> Research of all relevant programme and project documents as well as monitoring data,
> Semi-standardised online survey with project participants and their implementation partners as well as other potentially eligible applicants and
> Guided qualitative interviews with programme and project managers.
Evaluation of the admission procedures to scholarship programs of the Hans Böckler Foundation (HBS)
The Hans Böckler Foundation (HBS) awards scholarships to secondary school leavers, artists and students as well as to doctoral candidates in all academic fields. To obtain a scholarship, applicants go through a multi-stage and meanwhile digitalized admission procedure in which their appropriateness is assessed on the basis of a series of criteria. CEval GmbH was commissioned by the HBS to evaluate the application procedures. In particular, the CEval GmbH examines the appropriateness, purposefulness and efficiency of the procedures as well as the fairness and comprehensibility of the test criteria.
Therefore, the CEval team orientates itself on the OECD-DAC criteria (relevance, coherence, efficiency, effectiveness, impact, sustainability), which are used to systematically develop an analytical matrix. The evaluation follows a mixed-methods approach combining qualitative and quantitative instruments for data collection and analysis. Semi-structured guided interviews, qualitative short interviews, an online survey and a secondary data analysis will be applied.
Evaluation of the publication of the newsletter of the City of Saarbrücken
The state capital of Saarbrücken is required by law to publish public notices. In 2021, the city council of the state capital Saarbrücken decided to introduce the newsletter of the state capital Saarbrücken. The newsletter contains information on public announcements and tenders as well as other information from the administration. The newsletter is distributed free of charge to all accessible households (approx. 90,000) in the city of Saarbrücken.
The publication of the newsletter had been evaluated two years after its introduction. Among other things, it had been examined how well known the newsletter is among the citizens, whether the newsletter reaches the citizens’ households and by which means (print or online edition) this is done. For this purpose, about 800 citizens living in the city of Saarbrücken had been interviewed in short personal surveys in pedestrian zones and other busy places in various parts of Saarbrücken.
Data collection for the certificate “Fan project in accordance with the National Plan for Safety in Sports” (German: „Fanprojekt nach dem Nationalen Konzept Sport und Sicherheit“)
In Germany, there is a dense network of social work institutions in the field of adolescent soccer fans (“fan projects”) that work in accordance with the National Plan for Safety in Sports (German: Nationales Konzept Sport und Sicherheit). A certification of those fan projects has been launched in 2010. Offering fan projects an instrument of quality control, the certification is intended to pursue multiple goals: Establishing a common standard for fan projects, increasing confidence in the work of fan projects, and improving the working context of fan projects. The certificate is awarded on the basis of a recommendation by the Quality Management Group of the Centre for Fan Project Coordination of the German Sports Youth (German: Koordinationsstelle Fanprojekte bei der Deutschen Sportjugend). The Quality Management Group includes representatives from soccer, the highest state and family authorities, science, and fan projects themselves. After the certification was revised and tested by CEval GmbH in two previous projects and subsequently rolled out nationwide by collecting the information required for the certification in all fan projects, CEval GmbH has now been commissioned with the data collection for the third phase of the certification process for the current 71 fan projects.
Evaluation of a digital Senior Citizens Portal “digital.vital”
The senior citizens’ portal “digital.vital” is a model project of the Amt Hüttener Berge, the district of Germersheim and the city of Püttlingen. The aim of the portal is to make a contribution to meeting the challenges associated with demographic change and advancing digitalisation: Through needs-based offers, people are to be supported in the areas of nutrition, exercise and leisure activities/social participation (incl. housing, mobility, support) in achieving and maintaining a high quality of life in old age. At the same time, the portal also aims to strengthen the digital participation of older people. The project started in July 2020 and from autumn 2022 various services and information will be made available on the portal.
The Amt Hüttener Berge has commissioned CEval GmbH to evaluate the senior citizens’ portal in order to gain insights into the quality of the information and services offered and to identify possible potential for optimising the user experience in terms of portal functionality and content. Methodologically, a total of four participant observations will be carried out for the structured evaluation of the user experience among the pilot users in the context of so-called digital workshops at various locations. The participant observations will be supplemented by follow-up interviews with selected participants a few days after their visit to the digital workshops in order to collect data on the actual benefits of the workshops and the use of the website by the participants. In addition, a standardised online survey will be conducted among senior citizens’ organisations and actors in the partner municipalities on their perceptions and professional assessments of the senior citizens’ portal, its services and potential for further development.
Evaluation of the digital senior portal “digital.vital
he senior citizens’ portal “digital.vital” is a model project of the Amt Hüttener Berge, the district of Germersheim and the city of Püttlingen. The aim of the portal is to make a contribution to meeting the challenges associated with demographic change and advancing digitalisation: Through needs-based offers, people are to be supported in the areas of nutrition, exercise and leisure activities/social participation (incl. housing, mobility, support) in achieving and maintaining a high quality of life in old age. At the same time, the portal also aims to strengthen the digital participation of older people. The project started in July 2020 and from autumn 2022 various services and information will be made available on the portal.
The Amt Hüttener Berge has commissioned CEval GmbH to evaluate the senior citizens’ portal in order to gain insights into the quality of the information and services offered and to identify possible potential for optimising the user experience in terms of portal functionality and content. Methodologically, a total of four participant observations will be carried out for the structured evaluation of the user experience among the pilot users in the context of so-called digital workshops at various locations. The participant observations will be supplemented by follow-up interviews with selected participants a few days after their visit to the digital workshops in order to collect data on the actual benefits of the workshops and the use of the website by the participants. In addition, a standardised online survey will be conducted among senior citizens’ organisations and actors in the partner municipalities on their perceptions and professional assessments of the senior citizens’ portal, its services and potential for further development.
Evaluation of the prevention campaign “Mach Dein Passwort stark!”
In January 2019, the Ministry of the Interior of North Rhine-Westphalia (IM NRW) commissioned the NRW State Criminal Police Office (LKA NRW) to develop a prevention and publicity campaign to combat ‘cybercrime’. The aim of the campaign was to strengthen the population in NRW in the safe use of digital end devices and thereby reduce opportunities for cybercrime in the broader sense and make the work of the perpetrators more difficult. The cross-media campaign “Make your password strong!” was launched in October 2020.
CEval GmbH was commissioned to carry out an accompanying evaluation for the campaign with the two sub-areas of process and impact evaluation. The aim of the study was to evaluate the implementation process as well as the effectiveness of the “Make your password strong!” campaign and to derive conclusions and recommendations for future police prevention campaigns based on these results.
The evaluation was based on a broad mix of qualitative and quantitative data collection involving all actors involved in the design and implementation of the campaign as well as the target groups. The design also included an experimental trial set-up for comparative impact evaluation of the different components of the campaign. Descriptive and inferential statistical methods as well as qualitative content analyses were realised within the framework of the data analysis.
Evaluation of the “PROBONO Teacher Training” Program in Tanzania
The PROBONO Teacher Training program aims at contributing to the improvement of education quality in Tanzania as the results of national final exams indicate a great need especially in the STEM subjects. This is done by delivering a 4-year in-service capacity training in form of workshop activities, providing equipment and infrastructure in their partner schools and fostering a network of education authorities and actors.
The final evaluation was based on the OECD-DAC critieria (relevance, effectiveness, impact, sustainability) in order to derive recommendations for shaping the program´s next phase.
The evaluation utilized different methods: next to interviews with the leadership of the project, project partner and head of schools; Focus Group Discussions gave voice to teachers, students and trainers. A comprehensive survey among all participating teachers and trainers enabled an in-depth quantitative analysis.
Data collection for the certificate “Fan project in accordance with the National Plan for Safety in Sports”
In Germany, there is a dense network of social work institutions in the field of adolescent soccer fans (fan projects) that work in accordance with the National Plan for Safety in Sports (German: Nationales Konzept Sport und Sicherheit). A certification of those fan projects has been launched in 2010. Offering fan projects an instrument of quality control, the certification is intended to pursue multiple goals: (a) establishing a common standard for fan projects, (b) increasing confidence in the work of fan projects, and (c) improving the working context of fan projects. The certificate has been awarded by the Quality Management Group of the Centre for Fan Project Coordination of the German Sports Youth (German: Koordinationsstelle Fanprojekte bei der Deutschen Sportjugend). Members of the Quality Management Group belong to several stakeholder groups: soccer, federal authorities (for family affairs), science, and fan projects themselves.
During a first subproject (5/2015-6/2016), CEval GmbH (a) revised the certification process, (b) tested the revised certification process on five fan projects, and (c) developed a standardized, feasible concept for collecting information necessary for certification. Thereby the certificate attests fan projects high quality of work.
Within second subproject, CEval GmbH supported the nationwide roll-out of the revised certification by collecting all necessary information on all the remaining German soccer fan projects.
Impact analysis of the project “Kita im Aufbruch: Prozessbegleitung Richtung Nachhaltigkeit” [Kita im Aufbruch: Process support for sustainability in education]
The State Association for the Protection of Birds in Bavaria e.V. (Landesbund Vogelschutz, LBV) had implemented the environmental education project “Kita im Aufbruch: Process support for sustainability in education” between November 2020 and March 2022. The aim of the project was to contribute to the structural embedment of Education for Sustainable Development (ESD) in early childhood care by process support for selected Bavarian day-care centers. The CEval GmbH conducted an impact analysis of the project. By means of a before-and-after survey of day-care center staff, parents and day-care center operators, the impact of the project on the competencies of the pedagogical staff and the quality of educational offers as well as pedagogical processes in terms of ESD had been investigated. In addition, the CEval team further analyzed to what extent the project contributes to a holistic transformation of the learning and teaching environment towards a structural embedment of ESD in the daycare centers. In addition to pedagogical concepts, this also includes the areas of sustainable procurement and catering in the sense of a sustainable menu. In order to be able to clearly assign observed effects and impacts to the interventions of the project, the impact analysis was based on a quasi-experimental design including comparison groups (daycare centers without participation in process support) into the survey. The team combined, qualitative and quantitative survey methods in the process of data collection.
