Strengthening evaluation capacities within government and civil society in Uganda (ECD Uganda): Development of a blended-learning Master of Evaluation (2012-2015)
Evaluation as a scientific means to describe and assess interventions is gaining relevance worldwide. The needs for qualified evaluation professionals are increasing in Uganda, too. However, at the moment these needs cannot be satisfied due to lack of training- and evaluation programs in this region. It is for this reason that the BMZ together with CEval and GIZ initiated a program for developing evaluation capacities in Uganda.
The most important component of this program is the development and implementation of a master course in evaluation in the blended-learning format at Uganda Technology and Management University (UTAMU) in Kampala, Uganda. Blended-learning is an innovative concept which is gaining popularity worldwide and is now the default means of teaching at UTAMU. Blended-learning unites traditional face-to-face classes with modern forms of e-learning in a didactical coherent way. An advantage of this concept is, that it is possible not only for Ugandan students to attend classes but also for all students of English-speaking African countries.
The Master of Evaluation of Saarland University in cooperation with HTW Saarland serves as a template for the Ugandan Master, which will be introduced in fall 2015. The contents of the master program have been developed in cooperation with CEval GmbH, the teachers of the German master as well as Ugandan contact persons of Makarere University and have been adapted to the Ugandan context. The Ugandan Master of Monitoring and Evaluation will comprise 15 modules in total, seven of which are in compliance with the basic teaching program of evaluation at Saarland universities. This is to ensure that all topics relevant to acquire profound knowledge in the field of evaluation are covered. UTAMU has complemented this program with further modules regarding project management, monitoring as well as business start-up consulting. The modules are roughly comprised of web-based trainings, e-tutoring and face-to-face seminars. The teaching concepts have also been developed in cooperation with all stakeholders. The GIZ is responsible for the technical implementation of the e-learning concepts.
Two modules of the master of evaluation have been tested in an experimental phase together with UTAMU. During this experimental phase e-tutoring as well as the face-to-face seminars have been supervised in the tandem-principle by teachers of UTAMU and Saarland University/HTW. The hosting of the online study course has been undertaken by the GIZ platform GC 21. In a next step, the seven modules have been integrated into the already existing UTAMU master and the new course of study has been submitted to the responsible committees. The course of study is now aggregated and will be introduced experimentally. A decision about whether the course of study will be continued and which modifications would be necessary for this procedure, will be made in December 2016.
Constraints Faced by Graduates in the Labour Market of the Greater Region
The Unit for co-operation between Science and the Working World (KoWa) in cooperation with Arbeitskammer (Chamber of Labour) Saarland commissioned the research project Labour Market for Graduates in the Greater Region (Saarland and Rhineland-Palatinate in Germany, the Grand Duchy of Luxembourg, the French region of Lorraine and Belgian Wallonia). The intended use of this study was to provide findings about graduates labour market opportunities in Saarland, as well as about favorable and constraining conditions for graduates who wish to enter the border-crossing or the Saarland labour market. Thanks to previous research, it was known that graduates face considerable difficulties when trying to gain access to the border-crossing labour market, despite a number of specialized placement services or study programs which both involve several universities in the Greater Region.
Yet, an encompassing, representative survey among graduates of higher education institutions in Saarland, focusing on motives and backgrounds of the decision if or if not to search for jobs in neighbouring foreign countries during their transition into employment, has not yet been conducted. Also, there was hardly any information about those graduates from Lorraine, Luxembourg and Wallonia who search for jobs in Saarland. This study aimed at providing the lacking data by conducting a graduate survey among all students who graduated from Saarland University during the last eight years. Via email and via postcards sent to their home addresses, graduates had been invited to participate in an online survey. Additional graduate surveys at different universities of the Greater Region were being planned.
Evaluation of Staff/Management Talks at Saarstahl AG
In 2013, Saarstahl AG Management introduced Performance appraisals between Supervisors and their Managers (‘Mitarbeitergespräche’, short: MAGs) to the company’s human resource management. These annual talks take place in private, Independent of any particular incident or Event and on a non-hierarchical Basis. The aim of the MAGs at Saarstahl is to improve orientation towards the company’s Targets and its corporate culture, to promote employee Training and development and to improve cooperation within the company.
The evaluation of the 2013 MAGs was commissioned in order to Analyse if the implementation process had been successful. A special Focus wa on the following aspects: preparation of employees (seminars, information material); implementation and communication of target Agreements as well as employee support and development strategies; General manageability of this tool regarding Content and structure.
In close cooperation with Saarstahl AG, CEval developed and conducted a standardised online Survey among employees at all levels of the company. In preparation, guideline-based interviews were conducted with those members of staff responsible for the implementation process as well as with selected employees at different levels of the company.
Quality Management Support at Saarland Institute for Vocational Training and Further Education in Psychotherapy (SIAP)
The Saarland Institute for Vocational Training and Further Education in Psychotherapy (SIAP) was founded in 1995 and is part of Saarland Heilstätten GmbH (SHG; association providing a range of health care Services in Saarland). SIAP is a state-approved training centre, amongst others for Counselling with a Focus on psychological approaches as well as Children and Youth Counselling.
For both programmes, SIAP commissioned Consulting sservices, including an Evaluation, in order to a) identify potentials for improvement in teaching structures and b) develop a questionnaire for ongoing Quality assurcance.
Analyses were based on existing files and records, programme documents and a review of relevant literature, but mainly on guidline-based and group interviews with all relevant actors: students (at different stages of their Trainings), teachers for practical sessions, lectures and Supervisors.
Special Focus has been laid with the question if and how optimisation of teaching processes can take pressure of programme participants while simultaneously providing a targeted preparation for their future work in counselling. Encompassing systematic data analysis resulted in a Brief evaluation report including recommendations for programme development.
Subsequently, a questionnaire for ongoing Quality assurance of both programmes was developed which took results from previously conducted literature Reviews and qualitative interviews into consideration. Additionally, CEval developed a guideline which enabled SIAP to collect and Aanalyse data independently after the end of the evaluation project.
Evaluating the implementation of the Internet strategic concept “Country Web Portals as a Cross-cultural Internet Platform”
Object of the evaluation was the implementation of the
Internet strategy concept developed by the Goethe Institute (GI), known as Country Web Portals as a Cross-cultural Internet Platform. GI adopted this concept in 2011 and based its activities on it following the guiding principle that
– the personal experience of meeting and communicating through multimedia and online interaction is of importance,
– the platform offered online is characterised through (some) diversity and durability and
– users can participate and contribute content. Thus, the intercultural platforms target both the transfer of information as well as cross-cultural awareness and Exchange. The evaluation study examined whether the objectives of the country web portals have been achieved and the intended target groups have been reached. In addition, the analysis included the question on how effectively these web portals had enabled information transfer and promoted intercultural exchange. Also, the evaluation provided specific conclusions and recommendations derived from the collected information in order to further develop the concept or the country Portals. CEval followed a “mixed methods” approach with regard to data collection. As a result, existing documents and data derived from an online survey of the GI had been analyzed in detail. Additional surveys had also been conducted (qualitative guideline based interviews).
Process Evaluation of the Career Support Program ’Talente sichern & Zukunft gestalten’ (Securing Talents Shaping the Future)
Talente sichern Zukunft gestalten (Securing Talents – Shaping the Future) is a leadership development programme for young women run by Cusanuswerk – scholarship body of the Catholic Church in Germany. Mentoring and coaching by experienced academics and professionals as well as building a peer network constitute the main elements of this program.
Since its launch in 2007, the quality of the program had been assessed annually by means of external process evaluations conducted by the Center for Evaluation (CEval), and a similar evaluation had been carried out in the programs fourth year 2013-2014. This evaluation was designed to provide responsible members of staff at Cusanuswerk with information relevant to secure a successful performance adapted to the participants specific needs. Therefore, questions about overall satisfaction with offered courses and the quality as well as the professional and personal benefits of mentoring activities for both mentees and mentors were of particular interest.
The content of the existing and proven evaluation concept was sharpened as part of the evaluation by applying the CIPP (Context, Input, Process, Product) model, while ensuring comparability of evaluation results since the start of the career development program. Methodologically, the evaluation process was based on standardized online questionnaires, which mentees and mentors completed at three different stages of the program, the aim being to identify potential difficulties and appropriate solutions sufficiently early in the process.
Member Survey of IMMOEBS e.V.
IMMOEBS e.V. is the alumni and sponsor association of post graduate and master programs of real estate economics at the European Business School and the University of Regensburg. IMMOEBS was founded in 1991 and has about 2.600 members at the moment. It is the major alumni network of real estate economics in Germany.
CEval was commissioned by IMMOEBS with the conduction of a member survey. The general purpose of this online survey was the assessment of IMMOEBS and its activities by its members as well as the identification of potentials for development and adaptation requirements.
DAAD scholarship program ’Deutsche Auslandsschulen’
The DAAD scholarship program Deutsche Auslandsschulen (German schools abroad) exists since 2001; since 2008, it is integrated into the initiative Schulen: Partner der Zukunft (Schools: Partners for the future PASCH) of the Federal Foreign Office. The programme aims at offering the most talented and high-performing graduates from German schools abroad (including, since 2004, so-called Sprachdiplomschulen and since 2010, so-called Fit schools) scholarship-funded academic education prospects in Germany immediately after graduation, thereby tying them to the German academic landscape and economic market. In so doing, it is hoped that the scholarship holders and program alumni will act as multipliers for German and bilateral concerns in science, politics, culture and the economy.
Besides assessing the degree to which the program has achieved its objectives, the evaluation commissioned by DAAD addressed the appraisal of key programme parameters and program implementation. In this context, another key issue that had been addressed relates to the question to what extent the scholarship program was integrated into the PASCH initiative. Based on the evaluation results, recommendations for further program development and its integration with other DAAD and PASCH measures had been elaborated.
For data acquisition, CEval GmbH followed a mixed-methods approach, combining the analysis of program documents and statistics with a series of in-depth stakeholder interviews, several standardized online surveys and participant observation.
External auditing as part of the awarding of the quality seal “Fan project in accordance with the National Concept of Sport and Safety”.
In Germany, there is a dense nationwide network of social work facilities for young soccer fans in the form of fan projects, which work in accordance with the framework and specifications of the National Concept for Sport and Safety (NKSS). For the purposes of quality assurance and continuous improvement of the work of the fan projects, the fan project quality seal was introduced in 2010 in accordance with the NKSS. This is intended to contribute to an improved perception of the fan projects and to provide the fan projects with certainty of action. The quality seal was included in the revised NKSS in 2012 and is thus recognized at a high political level. The quality seal is awarded by the Quality Assurance Working Group, in which stakeholders from soccer, the highest state and family authorities, academia and fan projects are represented.
The award procedure for the quality seal should be revised from both a content and procedural perspective in order to create a seal that is a valid sign of the goodness and quality of a project. The award procedure should be as objective as possible and ensure transparency about the reasons for awarding or not awarding the seal. In the context of the revision of the seal, it was therefore the task of the Center for Evaluation (CEval) to develop a criteria catalog as well as a standardized survey questionnaire. In addition, the CEval, in consultation with the partners, developed a concept for the implementation of a pilot phase in which the revised quality seal and its awarding process were tested for their practical suitability. The pilot phase was implemented with five selected fan projects. Based on the findings and experiences of the pilot phase, the concept and the instruments were finalized.
University Cooperation with Universidad de Costa Rica (UCR), with Funding Provided by the DAAD Program Subject-Specific Partnerships with Universities in Developing Countries
Having completed a successful 3-year cooperation focusing on the further development of the postgraduate study programme Project and Programme Evaluation for Social Development established at Universidad de Costa Rica (UCR) in 1995, the Center for Evaluation (CEval) and the Universidad de Costa Rica (UCR) identified and realised new areas of cooperation. Since 2008, the German Academic Exchange Service (DAAD) has supported this cooperation with its funding project for sending German academics to universities in developing countries. Since 2012, the DAAD has also provided funding through the programme Subject-Specific Partnerships with Universities in Developing Countries (2012-2015), financially ensuring the plans to enhance the partnership between the two universities in Germany and Costa Rica.
The goal of the DAAD programme was to improve the teaching situation and to set up sustainable teaching structures in developing countries by facilitating academic exchange. Therefore, the international exchange between students, PhD candidates and lecturers was one of the most important aspects in the context of this enhanced cooperation. The focus also laid on opening up the master programme to other Central-American areas (regionalisation), embedding the partnership into an existing, international network (internationalisation), and adapting teaching of the master programme at UCR to regional demands and ensuing educational needs (creating a distinct programme profile). The four year partnership helped establishing or expanding regional and international professional networks and developing additional capacities regarding teaching and research.
It is planned to continue with the partnership in 2017 and integrate another Latinamerican university into the cooperation.
