Program Evaluation Overview
Quick Links
Getting Started
In late 2022, Water First embarked on a journey to develop and implement an evaluation framework to better understand our program strengths, challenges, and opportunities. The early stages of our evaluation work involved consultation with various collaborators, including Water First staff, management, and Indigenous Advisory Council members, as we worked together to map out our plan and approach.
We collectively determined that the overall goal of the evaluation would be to provide information to better support decision-making about program delivery, growth, management, and change, to demonstrate program effectiveness, and to better understand the extent to which each program addresses its primary goal. The primary goals associated with Water First’s three program areas all point back to our mission, which is to work in partnership with Indigenous communities to address local water challenges through education, training and meaningful collaboration.
Logic Models
Logic models were an important first step on the evaluation journey. They provided a visual roadmap where we could make connections between the resources used to operate each program, the planned activities associated with the program, and the anticipated outputs and outcomes. This process was informed by Water First’s “Framework for How We Work”.
For new and existing programs, we work collectively to identify each program’s primary goal. This step helped us better understand what each program is trying to accomplish, and what the program is doing to get there.
After developing a logic model for each of Water First’s three program areas, we developed key indicators to answer our guiding evaluation questions.
Clarifying program goals:
The logic model
An important first step for evaluation planning.
Inputs
Resources needed to operate each program area at Water First.
Activities
If we have access to these resources, then we can accomplish our planned activities.
Outputs
If we accomplish our planned activities, then Water First should be able to deliver the planned amounts of service intended.
Outcomes (short- and medium-term)
If we accomplish our planned activities, then Water First should be able to deliver the planned amounts of service intended.
Outcomes (long-term)
If benefits are achieved, then certain changes can be expected to occur over time. The intended result over time ties back to program growth, program goals, and long-term outcomes.
The primary goals associated with Water First’s three program areas all point back to our mission, which is to work in partnership with Indigenous communities to address local water challenges through education, training and meaningful collaboration.
Data Collection/Gathering
We adapted existing resources, and created new ones, to build a set of data collection tools that address our evaluation goals, answer the evaluation questions, and help us to track key indicators over time.
Water First surveys and community engagement tools are used to gather feedback from past and present community partners, interns, project participants, students, and staff. These data collection tools are consistently used at varying stages of program delivery. By honouring feedback from those invested in Water First programming, this allows us to continue to strengthen our relationships and deepen our understanding of community priorities, as well as program strengths, challenges, and goals.
Data Analysis & Application
With the information gathered through this process, we are better positioned to guide program delivery and expansion that is in line with community feedback and priorities.
The goals and activities associated with evaluation are as follows:
- Evaluate ability to meet goal(s), aligned with Water First’s vision and values.
- Demonstrate action toward planned outcomes and outputs, and identify key successes and gaps in program design and delivery.
- Interpret data collected and apply learnings to support strategies, evidence-based decision-making, program growth, and continuous improvement.
- Provide another measure of accountability to interest holders, partners, donors, and funders.
PROGRAM LOGIC FRAMEWORK
All primary goals are rooted in partnership with communities. Key indicators allow us to measure to what extent and how program goals are being met. Success is measured through each program’s ability to address community priorities, build meaningful relationships, and deliver education and training that has lasting results.
Drinking Water Internship (FY 2025 - FY 2026)
Program Goal / Objective
Summary | Indicators | Verification Sources | Assumptions / Risks |
|---|---|---|---|
Overarching goal: To serve any Indigenous community in Canada who sees value in Water First’s programs Program specific goal: To train Indigenous adults to become certified water treatment operators, in order to provide safe, clean drinking water for future generations. This goal is achieved through: Working in partnership with Indigenous communities to address local water challenges through education, training and meaningful collaboration. | % graduates @1 month employed in water treatment field % graduates @12 months employed in water treatment field % partners strongly agree/agree increased internship employment related to training % satisfaction with WF ability to collaborate with community % partners/participants would refer Water First to another Indigenous community | Follow-up check-in with graduates @1 month post-internship Follow-up check-in with graduates @12 months post-internship Community partner follow-up survey Pre-program and endpoint intern surveys | That Indigenous adults who complete Drinking Water Internship training will be interested in pursuing and/or be able to access employment as local water treatment plant operators. Local Indigenous community/Tribal Council support for the Drinking Water Internship. |
Purpose / Outcome(s)
Summary | Indicators | Verification Sources | Assumptions / Risks |
|---|---|---|---|
To what extent and how does the program work in meaningful partnership with Indigenous communities to: Provide technical training (build skills) to increase Indigenous-led resource management Improve access to local water science education and training opportunities Increase local employability in Indigenous communities within the water treatment sector | % interns strongly agree/agree internship helped develop skills to become a water treatment plant operator % partners strongly agree/agree internship strengthened local technical knowledge to deliver clean drinking water % increase in self-rated confidence across three core learning areas (Water Quality Knowledge and Application; Workplace Skill Development; and Career and Employment Skill Development) from internship start to finish % interns see themselves applying skills learned to pursue a career in the water sciences and/or in a water treatment plant | Pre-program and endpoint intern surveys Community partner follow-up survey | Regular internship attendance will help interns develop skills and technical knowledge needed to secure employment in the water sciences/water treatment sector. Collaborative partnerships between Water First and Indigenous community leaders effectively increase local technical capacity to manage and provide safe, clean drinking water. |
Outputs
Summary | Indicators | Verification Sources | Assumptions / Risks |
|---|---|---|---|
Interns graduate from a 15-month, paid internship program to become certified water treatment plant operators | # internships successfully delivered # interns recruited % intern retention rate % female intern graduates % of certifications per internship % increase in self-rated confidence across three core learning areas (Water Quality Knowledge and Application; Workplace Skill Development; and Career and Employment Skill Development) from internship start to finish | Quarterly review of program outputs Pre-program and endpoint intern surveys | Providing local, Indigenous-partnered, community-driven access to training means that local Indigenous adults will want to and/or be able to regularly attend, and complete the training and related certifications. |
Activities
Summary | Indicators | Verification Sources | Assumptions / Risks |
|---|---|---|---|
Launch 2 new internships (AKRC, SOI) using Tribal Council and Community models Improve regional focus: – Curriculum updated – Refine intern, operator, and local coordinator recruitment strategies Integrate new LMS Integrate evaluation feedback into program planning Increase Alumni engagement Increase interdepartmental collaboration | # interns recruited # potential partners identified across geographies # partners engaged % satisfaction with WF collaboration % curriculum adaptations developed for new regions % existing curriculum revised (based on learner feedback and engagement) Local Recruitment handbook complete and shared with partners % partners/WF DWI staff strongly agree/agree collaboration went well % evaluation recommendations integrated into program planning and adaptation % alumni recruited % alumni engaged in WF supports (i.e., professional development fund, networking) % alumni engaged in interdepartmental programming % interns strongly agree/agree interdepartmental collaboration provided an important opportunity to share knowledge with younger generation of learners (i.e., School Water program) | Quarterly review of program outputs Instructional Design & Development records Water First staff and management debrief sessions Partner and intern surveys Evaluation Action Items Alumni Network program and event tracking | New regional partners are available and interested. The community supports the training. Funding is available. Sufficient FTE are available. Sufficient resources (i.e., curriculum) are prepared and available. |
Environmental Water Internship (FY 2025 - FY 2026)
Program Goal / Objective
Summary | Indicators | Verification Sources | Assumptions / Risks |
|---|---|---|---|
Overarching goal: To serve any Indigenous community in Canada who sees value in Water First’s programs Program specific goal: To train and certify Indigenous adults in the environmental water science field, in order to increase technical capacity to manage land and water resources for future generations. This goal is achieved through: Working in partnership with Indigenous communities to address local water challenges through education, training and meaningful collaboration. | % graduates @1 month employed in environmental field % graduates @12 months employed in environmental field % partners strongly agree/agree internship increased access to employment opportunities related to training % partners/interns EW staff strongly/agree collaboration between WF and community partner(s) was successful % partners/participants would refer Water First to another Indigenous community | Follow-up check-in with graduates @1 month post-internship Follow-up check-in with graduates @12 months post-internship Community partner follow-up survey Pre-program and endpoint intern surveys | That Indigenous adults who complete Environmental Water Internship training will be interested in pursuing and/or be able to access local employment in the environmental water science field (i.e., Lands and Resources). Local Indigenous community/Tribal Council support for the Environmental Water Internship. |
Purpose / Outcome(s)
Summary | Indicators | Verification Sources | Assumptions / Risks |
|---|---|---|---|
To what extent and how does the program work in meaningful partnership with Indigenous communities to: Provide technical training (certifications) to build capacity in Indigenous-led resource management Improve access to local water science education and training opportunities Increase local employability in Indigenous communities within the environmental sector (i.e., Lands and Resources) | % interns strongly agree/agree internship helped develop skills needed to become an environmental technician % partners agree/strongly agree EWI training provided interns with skills and/or knowledge that have increased their capacity to work in local environmental sciences (i.e., Land & Resources office) % increase in self-rated confidence across core certification areas (Water Quality, Data, and Fish Knowledge, Skills, and Application) and Career and Employment Skill Development from internship start to finish % interns see themselves applying skills learned during the internship to pursue a new career and/or advance their existing career environmental sciences | Pre-program and endpoint intern surveys Community supervisor survey EW staff debrief process | Regular internship attendance will help interns develop skills and technical knowledge needed to secure and/or advance employment in environmental sciences. Collaborative partnerships between Water First and Indigenous community leaders effectively increase local technical capacity to increase capacity to manage local aquatic resources. |
Outputs
Summary | Indicators | Verification Sources | Assumptions / Risks |
|---|---|---|---|
Interns graduate from a 15-month, paid internship program to complete accredited certifications to build employable skills in the environmental water science field. | # internships successfully delivered # interns recruited % intern retention rate % female intern graduates % of certifications per internship % increase in self-rated confidence across core certification areas (Water Quality, Data, and Fish Knowledge, Skills, and Application) and Career and Employment Skill Development from internship start to finish | Quarterly review of program outputs Pre-program and endpoint intern surveys | Providing local, Indigenous-partnered, community-driven access to training means that local Indigenous adults will want to and/or be able to regularly attend, and complete the training and related certifications. |
Activities
Summary | Indicators | Verification Sources | Assumptions / Risks |
|---|---|---|---|
Complete Eco Canada accreditation process Launch 1 new internship? Revise curriculum based on pilot internship feedback from all interest holders Integrate all evaluation feedback into program planning Launch the Environmental Water Alumni Network in collaboration with DWI | % revisions completed and approved # interns recruited # potential partners identified # partners engaged % identified curriculum revisions implemented % evaluation recommendations integrated into program planning and adaptation % alumni recruited % alumni engaged in WF supports (i.e., professional development fund, networking) | Quarterly review of program outputs Instructional Design & Development records Water First staff and management debrief sessions Evaluation Action Items Alumni Network program and event tracking | New regional partners are available and interested. The community supports the training. Designated funding is available. Sufficient FTE are available. Sufficient resources (i.e., adapted curriculum) are prepared and available. |
School Water Program (FY 2025 - FY 2026)
Program Goal / Objective
Summary | Indicators | Verification Sources | Assumptions / Risks |
|---|---|---|---|
Overarching goal: To serve any Indigenous community in Canada who sees value in Water First’s programs Program specific goal: To engage Indigenous students, from K-12, to showcase pathways to careers in water science sectors. This goal is achieved through: Working in partnership with Indigenous communities to inspire future Indigenous water scientists through educational workshops and meaningful collaboration. | % students/staff strongly agree/agree SW programming increased knowledge of water science career pathways % satisfaction with WF ability to collaborate with community % partners/participants would refer Water First to another Indigenous community | Quarterly review SW team debrief documents Student surveys (school year, summer program) Teacher/staff surveys (school year, summer program) Community partner follow-up survey | That Indigenous students who participate in School Water programming will inspire lasting interest in pursuing water science career pathways. Indigenous community and local school/school board are interested in partnering with Water First to deliver SW program. |
Purpose / Outcome(s)
Summary | Indicators | Verification Sources | Assumptions / Risks |
|---|---|---|---|
To what extent and how does the program work in meaningful partnership/relationship with Indigenous communities to: Foster a love and increased understanding of water science Integrate local knowledge in program delivery (i.e., local Indigenous subject experts, Water First interns) Increase student awareness about water science careers. | % students/staff strongly agree/agree SW programming increased understanding of water science % partners strongly agree/agree SW programming increased students’ awareness of water science % students/staff strongly agree/agree SW programming increased awareness of water science careers % student engagement throughout the program | Student surveys (school year, summer program) Teacher/staff surveys (school year, summer program) Community partner follow-up survey SW staff debrief process | Regular workshop attendance will help students develop a deeper understanding of water sciences and water science careers. Collaborative partnerships between Water First, Indigenous community partners, and local schools will increase the number of Indigenous youth pursuing careers in the water science sector. |
Outputs
Summary | Indicators | Verification Sources | Assumptions / Risks |
|---|---|---|---|
K-12 students complete a 2-5 day program that fosters an increased understanding of water science career pathways. K-12 students who complete a set of 8-10 workshops receive a course certification. | # programs successfully delivered # students participating in program # workshops delivered # courses completed # high school credits earned (summer program) % students/staff strongly agree/agree SW programming increased awareness of water science careers | Quarterly review of program outputs Student surveys (school year, summer program) Teacher/staff surveys (school year, summer program) | Providing local, Indigenous-partnered, community-driven access to training means that local Indigenous adults will want to and/or be able to regularly attend, and complete the training and related certifications. |
Activities
Summary | Indicators | Verification Sources | Assumptions / Risks |
|---|---|---|---|
Implement Outreach Instructor model Improve quality of learning resources Increase school engagements Deliver 3 summer high school programs Integrate evaluation feedback into program planning Increase Water First intern/alumni engagement as subject matter experts in program delivery Increase interdepartmental collaboration | # Outreach Instructors leading program delivery % Outreach Instructors retained Impact on existing indicators % new curriculum content created % new curriculum content integrated # successful school engagements # high school programs % evaluation recommendations integrated into program planning and adaptation % interns/alumni engaged in SW programming % interns strongly agree/agree SW collaboration provided an important opportunity to share knowledge with students | Quarterly review of program outputs Curriculum development records Water First staff and management debrief sessions Evaluation Action Items Intern surveys | New community and school partners are available and interested. The community supports program delivery. Funding is available. Sufficient FTE are available. Sufficient resources (i.e., adapted curriculum, equipment) are prepared and available. |
