Evaluability Assessment of UNICEF Jordan Country Program 2023-2027 commissioned by the UNICEF JCO
The purpose of the evaluability assessment of the UNICEF Jordan Country Programme (JCO) was to ensure that medium- and long-term results can ultimately be demonstrated. The scope of the evaluability assessment covered JCO Country Programme Document (CPD) 2023-2027, including its six programme sections (health & nutrition, education, child protection, adolescent & youth, WASH, social policy & protection), monitoring systems, annual planning, and evaluation plans and the cross-cutting priorities (gender equality, disability inclusion, climate change, social equity, social behaviour change (SBC) and protection from sexual exploitation and abuse (PSEA)).
The objectives of the evaluability assessment were (1) to provide an independent assessment of how well JCO is prepared for a country programme or strategic framework evaluation and (2) to provide recommendations to enhance the evaluability of the CPD. The aim was to examine whether JCO has planned to effectively manage key risks, to establish mechanisms for timely implementation of evaluation recommendations, and whether it possesses the essential resources and infrastructure required to prove results achievement. The EA aimed to enhance JCO’s preparedness for the upcoming evaluation of the CPD.
The evaluability assessment employed a theory-based approach to analyse the methodological preconditions necessary for assessing effectiveness, impacts, and sustainability of UNICEF’s Country Programme. These preconditions included clear definitions of program results, baseline data availability, and timely target data. The assessment also aimed to establish meaningful hypotheses to understand the causal relationship between the JCO and observed development results. Contextual factors were considered crucial for a valid analysis, with an emphasis on identifying and monitoring risks and framework conditions. A detailed evaluability matrix guided the systematic analysis and structured data collection. A mixed-methods approach was adopted, using primary and secondary data sources as well as qualitative and quantitative methods to ensure validity and reliability. Stakeholder participation was integral, involving interviews with heads of JCO units and sections, consultations, and feedback loops to incorporate diverse perspectives.
Development of an M&E concept for the BBK’s international project work
The International Affairs Division at the Federal Office of Civil Protection and Disaster Assistance (BBK) develops projects abroad as part of Germany’s international engagement. These are based on expertise in civil protection and disaster relief and are implemented in cooperation with national and local authorities abroad.
The BBK commissioned CEval GmbH to support the development of a needs-based monitoring and evaluation (M&E) system for international project work. The objectives of the assignment were: (i) the development of an M&E concept for the BBK’s international project work, taking into account existing project documents, the (political) framework conditions and the BBK’s strategic orientation as well as staffing capacities; (ii) the further development of existing instruments and the development of additional instruments to make the impact-oriented monitoring of international projects more efficient; (iii) the presentation of the concept and the instruments at a workshop for the BBK’s project staff and project managers.
In developing the M&E system for the BBK’s International Affairs Division, CEval GmbH pursued a theory-led approach in order to create the foundations for an impact-orientated M&E system. In addition, the team pursued a participatory approach with the close involvement of the client in order to best align the M&E system with the client’s needs. The team advised and supported BBK in the introduction of SMART indicators at various levels for a methodologically sound M&E system and organized an M&E workshop for project staff and project managers for the projects in Jordan, Ukraine and Tunisia.
#eSkills4Girls Initiative
The #eSkills4Girls initiative aims to close the digital gender divide by promoting the access of women and girls to opportunities to help improve their digital skills. The initiative was launched in 2017 by the BMZ under Germany’s G20 presidency and managed by the GIZ sector programme education (SP Education). Since then the initiative established partnerships with private and public sector actors and joined EQUALS, the global multi-stakeholder partnership promoting gender equality in the digital age. To improve the digital skills of women and girls, the initiative established the #eSkills4Girls Fund, which provided small grants to 27 existing grassroots initiatives through EQUALS. Additionally, the initiative served as community manager for 9 bilateral projects financed by the BMZ under the #eSkills4Girls banner. Other activities implemented by the initiative aimed at raising the awareness of policymakers, showcasing female role models, and fostering sharing and learning among projects.
CEval GmbH was commissioned by GIZ SP Education to conduct an impact evaluation of the #eSkills4Girls initiative. The goal of the evaluation was to understand how and to which extent the different activities achieved their respective goals. Based on the evaluation questions, the initiative was assessed against the OECD-DAC criterion ‘effectiveness’. To fill missing information and data after an initial review of existing secondary documents, online-based questionnaires for the sub-grantees of the #eSkills4Girls Fund were disseminated. Furthermore, project reports and monitoring data by bilateral project partners were reviewed. In addition, focus group discussion and interviews with representatives of the sub-grantees and bilateral projects were conducted to complete the qualitative data collection.
Evaluation of the Deval Funding Programme Rigorous Impact Evaluation
The RIE funding programme supported a matchmaking between actors from science and development cooperation, which led to funding of nine rigorous impact evaluations (RIE) to be implemented between March 2023 and September 2025. Scientific institutions in Germany were eligible to apply together with a development project funded by the Federal German Ministry for Economic Cooperation and Development (BMZ). The programme aims to remove barriers to the implementation of RIE and to support systematic anchoring of RIE in German development cooperation.
CEval GmbH was commissioned with the evaluation of the RIE funding programme by the German Institute for Development Evaluation (DEval). The evaluation is carried out according to the OECD-DAC criteria and is based on a comprehensive analysis grid. A mix of methods is applied: The RIE project proposals and interim reports are assessed by means of a systematic review, which is structured by an assessment grid. Guideline based interviews are conducted with all relevant stakeholder groups of the programme at two points in time. In addition, a standardised online survey is implemented, which addresses all scientists and representatives of the development organisations as well as local scientists in the partner countries who are directly involved in the implementation of the nine RIE.
In addition, a Delphi survey is implemented as innovative method for looking to the future. For the first Delphi round, international experts will be recruited and their perspective on the future role of RIE in German development cooperation will be examined by thematic-focussed interviews. The results will be assessed and returned to the panel participants in a second round of inquiry in the form of an online survey in order to identify, weight and prioritise converging and diverging perspectives and views of the experts.
Central Project Evaluation: Agribusiness Facilities for Africa
The project Agribusiness Facilities for Africa (ABF) serves as a pan-African knowledge and advisory hub on agribusiness trainings and has worked with about 30 countries across the continent. With the objective of improving the effectiveness of private and public agribusiness support measures and supporting organisations for micro, small and medium-sized enterprises in Africa, ABF develops training concepts for farmers and processors to meet their partners’ specific needs. The Farmer Business School (FBS) training teaches farmers and producers principles of agriculture as a business and promises to strengthen the management skills of MSMEs for better and diversified incomes and nutrition. Other training modules approaches offered by ABF include, Processor Business School (PBS), Cooperative Business School (CBS) and the more recently developed Aquaculture Business School (ABS ).
The evaluation was conducted according to the specifications and guidelines for central project evaluations of GIZ, which include a standardized analysis grid (evaluation matrix) based on the OECD-DAC criteria (relevance, coherence, efficiency, effectiveness, impact, sustainability). The evaluation team applies qualitative methods for data collection and analysis. Semi-structured interviews and focus groups discussions (FGDs) are conducted for primary data collection. During a 3 week field study in Uganda, Zambia and Malawi, the evaluation team will visit partner projects of ABF to assess the implementation and the impact of FBS, CBS and ABS teaching approaches on the ground. Qualitative content analysis will be applied based on written transcripts of the interviews, FGDs and the vast body of project documents. Causal relations between project activities and potential impacts, are assessed following a contribution analysis approach.
Evaluation of the Diakonie Katastrophenhilfe Programme Indonesia 2018-2023
The DKH Indonesia programme was launched in the immediate aftermath of three large scale natural disasters occurring in Lombok (earthquake), Central Sulawesi (earthquake and tsunami) and the Sunda Strait (tsunami) in 2018. The programme was composed of eight individual projects implemented by two Indonesian partner organisations in five provinces and featured immediate emergency response measures, recovery assistance and Disaster Risk Reduction activities with a focus on inclusion of people with disability and prioritisation of most vulnerable groups. CEval was commissioned to provide an independent assessment of the programme based on the OECD-DAC criteria.
The evaluation was based on extensive stakeholder participation as well as on a mixed-methods and theory-based evaluation approach. The evaluation team developed a comprehensive Theory of Change (ToC) at programme level and a detailed evaluation matrix. A desk-based systematic review of project documents was complemented by a three-week on-site study in Indonesia. The systematic review was based on a standardised assessment grid developed from the evaluation matrix and applied to aggregate monitoring data and relevant information from project documents. During the on-site study, the evaluation team conducted 29 interviews and 29 focus group discussions with programme and project staff, local authorities, and beneficiaries. Primary data was analysed applying qualitative content analysis. Findings were presented in a comprehensive evaluation report and during several workshops.
System Evaluation of the Evaluation System of the ‘International Cooperation’ of the Physikalisch-Technische Bundesanstalt
In 2021, CEval GmbH was assigned by the Physikalisch-Technische Bundesanstalt (PTB), the National Metrology Institute, with a system evaluation. The overall objective was a strategic improvement of the PTB’s evaluation system. Relevant points of the analysis grid and the evaluation questions were:
1. Evaluation policy, goals, strategies and criteria of evaluation.
2. Planning and procedure of evaluations, control processes, forms and scope of evaluations, financial resources, relation to the total budget
3. Standards of evaluation, quality assurance system, impact orientation, methodological quality
4. Anchoring of evaluation in the organization, independence of evaluation, credibility of evaluation
5. Personnel capacity, qualification, evaluators, further training, networking with the professional community
6. Utilization of evaluation results (processes of utilization, products of evaluation, dissemination of results, knowledge management system).
Methodologically, various data sources, such as documents and evaluation reports, were included and 28 guided qualitative interviews, as well as a group discussion, were conducted.
Pilot DC-Program Evaluation Sustainable Economic Development – Promotion of Youth Employment and Vocational Training in Kenya
In the future, evaluations of development cooperation (DC) programs should provide policy-relevant findings, especially on the establishment and coordination of DC projects and their interaction in a sector or region. In order to develop an appropriate format for DC program evaluations, experience was gathered during the pilot evaluation of the DC program “Sustainable Economic Development – Promotion of Youth Employment and Vocational Training in Kenya” and subsequently systematically processed.
The evaluation used a theory-based approach to empirically test assumptions about causal relationships between the services provided, the goals of the individual projects, and the overall goal of the program. The results are based on individual and group interviews with program and project managers, partner organizations and target group representatives. Furthermore, the program and project reporting was subjected to a qualitative content analysis.
Developments in fair trade and fair procurement in Germany 2015-2022. Follow-up to the study “Does fair trade change society?”
In 2015, leading stakeholders of the German fair trade movement commissioned a study on the impact of fair trade in Germany, which was carried out by CEval GmbH. The study investigated the impact of fair trade in four societal areas, i.e., civil society, consumers, retailers and producers, and politics and the public sector. Since 2015 the fairtrade sector as well as the framework conditions in which it operated have changed. This includes, in particular, the environment for a fair public procurement.
Against that backdrop, Fairtrade Germany, Engagement Global – Service für Entwicklungsinitiativen (Service for Development Initiatives) with its Servicestelle Kommunen in der Einen Welt (SKEW – Service Agency Communities in One World) and in cooperation with Forum Fairer Handel (Forum Fair Trade) and Weltladen-Dachverband (umbrella organization of German world shops) mandated CEval GmbH with the implementation of an updated and extended version of the 2015 study. The objective was to investigate the impact of fair trade in Germany between 2016 and 2022 under the changed framework conditions and taking into account the increased role of the German municipalities for fair trade and fair public procurement. Based on the findings, recommendations on how to further increase the effectiveness of fair trade and how to further support a fair public procurement have been put forth.
Methodologically, the study built on the approach developed for the 2015 study and relied on qualitative contribution analysis. On the basis of a so-called theory of change and its underlying hypotheses, the study investigated the changes that are intended by the fair trade community in the various areas of observation as well as the contributions that have been made by the fair trade community to these changes. In doing so, a mix of qualitative and quantitative methods of data collection and analysis was applied. The starting point was an extensive desk study of existing literature and secondary data, which was complemented by empirical data collected by means of guided key informant interviews, (focus) group discussions and a semi-standardized online survey.
Central Project Evaluation: Sector Programme Soil Protection, Combating Desertification, Sustainable Land Management
The Sector Programme Soil Conservation, Combating Desertification, Sustainable Land Management advised the Ministry for Economic Cooperation and Development (BMZ) on the topics of sustainable land management, soil conservation and soil related climate impacts. In addition, the project tested and implemented approaches for the utilization of soil conservation measures for climate protection with partners in India and Kenya. This included the implementation of a carbon certification system in Kenya and studies on the climate impact of sustainable land management methods in India. The experience and knowledge gained from these studies and pilots were to be disseminated at national and international levels.
The evaluation was carried out according to the specifications and guidelines for central project evaluations of the GIZ, which included a standardized analysis grid (evaluation matrix) based on the OECD-DAC criteria (relevance, coherence, efficiency, effectiveness, impact, sustainability). During a ten-day on-site mission in Kenya, the evaluation team conducted interviews and focus group discussions with stakeholders and beneficiaries of the carbon certification scheme. The data collection in India was implemented remotely (online interviews). A contribution analysis approach was applied to examine causal relationships between project activities and potential impacts.
