EMPOWER Project mid-term Evaluation Terms of Reference
Dates: 11th July 2019
Evaluation manager: Consortium M&E
based: Juba –South Sudan
Background on the context
South Sudan faces ongoing conflict since December 2013, with an escalation in the conflict since July 2016, economic collapse and disruption of livelihoods as well as a worsening, severe food insecurity. The last Annual Education Census (AEC) which, was completed in 2016 and encompassed data from 2015 and 2016, was conducted across 18 states in South Sudan. Conflict and insecurity prevented data collection in a further 10 states. The AEC indicated the existence of 24 TVET centres, of which the government managed 13 and 11 which were non-governmental (private, community or religious based or run by NGOs). 5,178 pupils, equating to 0.4% of all students in school in South Sudan, were enrolled in TVET centres, of which 3,462 were male (67%) and 1,716 were female (33%). The General Education Strategic Plan (GESP) 2017-2022 provides information on the number of technical secondary schools (public and private) with the 2016/2017 baseline indicating the existence of only five government TVET centres, and one private TVET centre, across the whole country. Thus, the number of formal TVET centres decreased by three quarters (from 24 to 6) within a two-year period (2015-2017). Information on the number, location and functionality of non-formal TVET centres offering short technical and vocational courses (of 3, 6 and 9 months’ duration) which take place outside of formal TVET secondary schools and tertiary institutions is not included in the 2017-2022 GESP.
Intervention specific to the evaluation
EMPOWER Project is implemented through a consortium of six organizations Norwegian Refugee Council (NRC), BBC Media Action, Fin Church Aid (FCA), Vocational and Skills Development Organization (VOSDO), Association of Christian Resource Organisation Serving Sudan (ACROSS) and Nile Hope, with NRC as the lead agency. The project is funded by the European Union ( EU) and is focusing on Strengthening civil society and enhancing technical and vocational education and training (TVET) for youth employment in South Sudan. The target locations of the TVET projects are former Warrap State, former Western Bahr el Ghazal State, former Jonglei State and former Central Equatoria State. Key stakeholder for this project are the two primary government ministries responsible for supporting TVET (Ministry of General Education and Instruction and the Ministry of Labour, Public Service and Human Resource Development), local authorities, UNDP, UNESCO, other civil society organizations involved in TVET/ youth empowerment and the private sector actors in South Sudan among others. The project aims to improve TVET standards as well as support in harmonization of TVET practices across the project locations, including harmonization of TVET syllabi, renovation and equipping of selected TVET centres, so that they may provide TVET services. During the duration of the project, it targets to; enroll 4,000 youth aged 15 to 35 years in quality TVET; engage 121 TVET teachers in provision of TVET services; engage seven Trainers of TVET teachers; recruit16 TVET centres Head Teachers ; train mentor 90 Centre Committee (CCs) members so that they may support the running and management of TVET centres; engage 13 radio stations to broadcast TVET content; support the establishment of 391 business run by youth thereafter support and mentor these businesses; support400 youth to engage in seed production businesses; capacity strengthening of local CSOs in five former states, through subgrants, trainings and mentoring.
Purpose of the evaluation and intended use
2.1. Overarching purpose
The purposes of the Mid-term evaluation are;
To inform principally EMPOWER Partners and the donor, but also other project stakeholders and partners on the progress in implementation, i.e. the delivery of planned outputs and outcomes achieved as well as progress towards achievements of the respective objectives, while highlighting enabling factors and challenges encountered. The mid-term evaluation also seeks to assess the coordination, management and cooperation among the consortium members as well as the established coordination/management structures.
To independently assess the relevance, effectiveness, efficiency, and sustainability of the project to date;
To provide evidence-based recommendations and lessons learnt to improve program design/strategies and update the log-frame, if and where necessary;
To inform resource allocation for a future round of proposals, as well as to inform decisions on scaling-up, replication, and /or continuation
To assess the coordination mechanism in place, management and cooperation among the consortium members
2.2. How will the evaluation be used?
The evaluation will be used by;
Considering and implementing recommendations and lessons learnt.
Building on and replicating best practices to project sites, sharing them with partners as appropriate.
Reviewing implementation plan and budget in line to recommendations provided in report.
Addressing common challenges in implementation across the consortium partners
2.2. Who will it be used by?
The mid-term evaluation will be used by the following;
- EMPOWER partners
- NRC to identify which partner requires more support/attention
- Relevant line ministries
Scope and lines of inquiry
Dates of the evaluation: 30th July 2019
Geographical coverage and targeted beneficiaries**
Partner: ACROSS in Centeral Equatoria, BBC MEDIA in Centeral Equatoria, Jonglei, Western Bar El Gazel, Pibor, Twic East, Bor Town, Duk and Juba, FCA, in Centeral Equatoria, Jonglei, Pibor, Bor, Fangak and Juba City, Nile Hope, Jongeli, Twic West and Duk, VOSDO, Western Bar el gazel, Warrap|Kangi, Gogrial West/south
NRC, Western Bar el gazel, Warrap, central Equatoria, Northern Bar el Gazel|Juba City, Wau town,Bor town,Wunrock, Mayen abun, Yei, Aweil East and Aweil town
Project: Strengthening civil society and enhancing technical and vocational education and training (TVET) for youth employment in South Sudan
Donors: European Union
3.2. Lines of inquiry
The line of inquiries will be linked to the following Mid-term evaluation objectives as described below
Review to what extent the EMPOWER project “Promote active citizenship and the empowerment of local CSOs” has delivered effective, efficient, relevant and timely activities to beneficiaries as set in the project log frame.
The Mid-term evaluation should assess the following:
To what extent have the planned objectives in the project log frame been reached, per indicator, disaggregated by gender and age and, when appropriate, by residential status?
To what extent are the project activities contributing to the overall project goal? Was the project effective in increasing skills of TVET graduates to improve their living status and responding to humanitarian needs?
What were the major factors influencing the achievement of the objectives of the project?
What opportunities for collaboration have been utilized and how have these contributed to the project effectiveness?
Have proper accountability and risk management framework(s) been in place to minimize risks on program implementation?
How effective has the consortium approach, including the related structures been in delivery of the project?
How efficient was the delivery of project by EMPOWER Partners not only in terms of expenditure, but also in terms of implementation of activities and delivery of outputs?
Was the project activity implementation (modality) cost-efficient, while not compromising quality?
What would have been opportunities within the project implementation to reach more beneficiaries with the available budget or reduce costs while reaching at least the same number of beneficiaries without compromising quality?
What choices were made in terms of collaboration and non-collaboration during program design? Why were these choices made?
Were alterations made to the program design in terms of collaboration during the implementation phase based on the reality on the ground?
What were the outcomes of these choices for effective and efficient program implementation?
How relevant were the objectives and activities, implemented by the project, in addressing humanitarian needs in the states mentioned above?
How do beneficiaries perceive the relevance of the project and how have the activities implemented improved their lives? Are there any successful stories of change?
How has the collaboration between NRC, Consortium partners, CSOs and line ministries contributed to appropriate response of specific needs and priorities of the beneficiaries?
To what extent was the project able to adapt and provide appropriate responses to context changes and emerging local needs, and the priorities of beneficiaries?
Was the project design timely in responding to the needs on the ground?
Were the activities timely implemented when compared to the project work plan?
Were funds available in time during implementation of the activities to respond to new developments?
To what extent has the collaboration between NRC, Consortium partners and line ministries and other stakeholders contributed to efficient and timely coordination of logistic activities and processes?
To what extent have the project beneficiaries been reached and what mechanisms were in place to improve coverage?
The Mid-Term evaluation should assess the overall quality of the implementation. It is important to include beneficiaries’ opinions on the quality of the services received.
What mechanisms have been in place to track implementation of the EMPOWER project? (i.e. internal monitoring, evaluation, accountability, learning (MEAL)) and quality assurance mechanisms)?
What mechanism did EMPOWER partners have in place for beneficiaries to give feedback and complaints? Are beneficiaries aware of their entitlements in the EMPOWER project? How are they involved in the project?
Did the quality of the outputs delivered by the project meet the needs and expectations of the beneficiaries? What do beneficiaries feel could be improved for enhancing local capacity?
To what extent have project outputs contributed to build long-term community capacity?
To what extent the project was participatory throughout the project cycle?
Assess whether the collaboration between NRC, EMPOWER Partners, Ministry of General Education and Instruction (MGEI) and the Ministry of Labour, Public Service and Human Resource (MoLPHRD) and Ministry of Youth has added value to the interventions with a positive effect on beneficiaries and other stakeholders. What has contributed to this added value and what has not?
Which of the interventions, approaches, modality/strategy have been most effective according to EMPOWER and line ministries (Ministry of Labour, Ministry of Education, Science and Technology)?
Is there any substantial evidence on how project learning were generated and applied to improve the implementation, effectiveness and efficiency of the project?
Who benefited from shared learning experiences (e.g. joint meetings, joint field visits, workshops provision on best approaches and methodology), mainly the NGOs or also the local sector, community members and beneficiaries?
How did the different actors learn from these experiences?
Complementarity and harmonization:
Are there any concrete examples of successful models of collaboration of EMPOWER with other partners on geographic level, not just in terms of avoiding duplication but increasing complementarity and integrated programs affecting the reach and impact on beneficiaries?
What are barriers and/or enablers to this?
To what extent were the activities of the project complementary to the work of other stakeholders, i.e. prevented duplication and contributed to the larger response activities in South Sudan?
How visible is European Union emblem, EMPOWER and partner’s logo visible at project sites.
What measures have been taken to create visibility of the project’s added value towards government line ministries and other INGOs.
What joint activities have been undertaken during the implementation of project, both at a field level as well as at a country office level? (Please provide concrete, short, substantial, cases of evidence)
Identify and assess key lessons learnt, challenges and draw recommendation for future programming of Technical Vocational Education and Training in South Sudan.
The Mid-term evaluation should at least include one lesson learned and recommendation per evaluation category, i.e. effectiveness, efficiency, relevance etc
What are the key lessons learnt so far per project objective? To what extent has the delivery of response activities contributed to effective, efficient, relevant and timely delivery of aid and enhanced impact for the beneficiaries?
What strategies are in place for the sustainability of the project? Are the strategies being put into action.
What mechanisms have EMPOWER and partners put into place in order to sustain the key programme Outputs and Outcomes?
How has the programme worked with local partners to increase their capacity in a sustainable way?
What motivations /mechanisms exist for partners to continue playing these roles?
What are the risks facing sustainability of programme Outputs and Outcomes?
To conduct Tracer Study of the TVET graduates
The tracer study should assess the following
To what extent have the skills gained by TVET graduated been (or not) utilized? What are the main reasons for not utilizing the skills?
Are there any concrete examples that skills gained improved graduate economic status?
How was the training delivered? Are graduates satisfied with quality of training?
How long does it take the trained youth to find employment?
The methodology will be developed by the consultant, as well as all relevant tools and present in the inception report. The four strategic objectives mentioned in chapter 2 should be assessed, including all research questions under each objective.
The data collection should include the use of a number of approaches to gain a deeper understanding of the outcomes of the project, including:
Desk review of background documents (project document, project monitoring data, progress report, field visit reports etc.).
Key informant interviews (e.g. with EMPOWER staff members, i.e. program manager, technical field staff, field coordinator, key community member and representatives from the line ministries) to gather substantial subjective evidence on the effectiveness, efficiency, relevance and timeliness of the project activities implementation and delivery
Focus group discussions (e.g. with stakeholders, community members, TVET undergraduates and Centre committees, The FGD will serve as input for the narrative subjective evidence.
Survey Application of structured survey questionnaires with a representative, random sample of target population to quantitatively assess outcomes. This will be in greater scope, breadth and depth compared to standard routine project monitoring
A learning event to disseminate lessons learned through a presentation and a workshop facilitated by the consultant.
Next to the data collection method, an appropriate and strategic sampling method should be selected e.g. snowball sampling, purposeful random sampling or mixed purposeful sampling methods.
Evaluation follow up and learning
The findings will be used to;
Give meaningful feedback to partners about the progress of the EMPOWER project, that is to say, Providing honest feedback - positive and negative for improvement.
Check-in point if the project objectives are still on track . Often as year progresses, objectives are diverted.
Address difficult issues which the report has reflected .
A management response will be developed within one month of the evaluation report being finalized. This will be followed up and tracked by EMPOWER consortium management.
A dissemination plan will be developed to ensure that important learning is shared with internal and external stakeholders
Management of the evaluation
The person responsible for ensuring that this mid- term evaluation takes place is the consortium MEAL/COM team leader. The position has been appointed to internally coordinate the process and will be the Mid-term evaluation team’s main focal point.
An evaluation Steering Committee (SC) will be established by EMPOWER team with the following members:
Steering Committee chair:
Steering committee members
The Steering Committee will oversee administration and overall coordination, including monitoring progress. The main functions of the Steering committee will be:
Establish the Terms of Reference of the evaluation;
Review and comment on the inception report and approve the proposed evaluation strategy;
Review and comment on the draft evaluation report;
Establish a dissemination and utilization strategy
Deliverables and reporting deadlines
The consultant is expected to lead, accomplish and submit the following deliverables within the agreed timeframe and budget:
An inception report, which will serve as an agreement between parties on how the evaluation will be conducted. Items to address:
Understanding of the issues and questions raised in the ToR
Data sources; how to assess the questions in the ToR
Research methodology, including suggested sample and size
Schedule of activities and traveling (timeline)
Proposal for a learning event/validation of evaluation findings.
Appropriate validated draft data collection tools (e.g. methodological guidelines, group interview questions)
Statistical packages to be use and data protection.
A max 35-page draft and final evaluation report (in MS Office and PDF for final), excluding annexes and in English.
A max of 20-page draft and final tracer report (in MS Office and PDF for final), in English.
Both reports should be in the format indicated below, to be submitted to NRC (EMPOWER-CMU) South Sudan Country office. It is preferable to illustrate the results by appropriate graphs, visuals, tables and/or a dashboard with an accompanied explanatory text. The report should consist of:
Executive Summary in bullets (max. 2 pages)
Methodology, including sampling and limitations
Analysis and findings of the evaluation. The analysis should be done according to the mid-term evaluation objectives.
Objective 4, include an implementation strategy for the recommendations
Address concerns, lessons learned and comments from EMPOWER partners.
Stories of change and quotes from respondents
Conclusions for each of the mid-term evaluation objectives
Recommendations for future projects
Relevant maps and photographs of the evaluation areas where necessary
Bibliography of consulted secondary sources
Finalized data collection tools
List of interviewees with accompanying informed consent forms
PowerPoint presentation of preliminary findings to EMPOWER
Learning event for project stakeholders, partners. The structure and activities of the learning and evaluation meeting(event) will be agreed with NRC (EMPWOER)
The data collection phase in the field is to be confirmed between the consultant and EMPOWER, but ideally would start in second week of August 2019 , with the final report deadline to EMPOWER-CMU by the first week of September 2019.
Evaluation consultant team
At least Master’s degree in Education, Vocational studies, International Development Studies, or a related field
At least 5 years’ experience in working with humanitarian sectoral programs relating to Technical Vocational Training) and civil society organizations (CSO)
Demonstrated experience with quantitative and qualitative research, data base management and statistical data analysis
Experience of working in South Sudan will be added advantage.
Experience of evaluating TVET programs
Proven record of communicating with beneficiaries.
Understanding of TVET policy, standards and protocols.
Ability to deal with hardship and remote area field work
Strong understanding of humanitarian and evaluation ethics and a commitment to ethical working practices
Experience of working in insecure environments in South Sudan and managing security risks
Action-oriented and evidence based approach and strong drive for results;
Highly developed self-management, and communication skills;
Application process and requirements
Along with CV interested candidates should submit;
A technical and Financial proposal explaining their comprehension of the ToR and how they would approach this assignment, summarising the methodologies and approaches they plan to use, including a timeline. NOTE: ONLY provide professional cost for the services. NRC will cover Visa, Flights, accommodation and locally hired data collectors’ costs**
Two samples of similar previous assignments.
Company profile or CV including three reference
All technical and financial proposal should be submit to [email protected]
Before 30th July 2019.