Organization and Background
Tamrack, Inc. (a pseudonym) is a national provider of cleaning supplies to hospitals and nursing homes. Founded in 2001 as a regional company, it has grown to operate across five states with headquarters in Boise, Idaho. Since the onset of COVID-19, Tamrack has moved progressively toward remote and telework models to improve employee satisfaction, work-life balance, and talent acquisition. The company currently onboards 30 or more new employees per year and expects that number to grow.
Program and Stakeholders
In 2020, Tamrack transitioned its New Employee Onboarding (NEO) Program to a fully online format. The program combines pre-boarding tasks, day-one Zoom sessions with managers and mentors, and self-paced eLearning courses covering soft skills, compliance, and job-specific content. Its primary goal is to prepare new hires to feel confident and competent in their roles while fostering a sense of belonging within the organization.
Two upstream stakeholders drove the evaluation request: Bill Berg, HR Director, and Gill Gibson, Director of Learning and Performance Solutions. Managers of new employees served as key downstream contributors, alongside past program participants whose survey data informed the findings.
Evaluation Request
As Tamrack prepares to transition all branches to online-only onboarding, Berg and Gibson requested a formal evaluation to assess whether the NEO Program was effectively meeting its goals and to identify opportunities for improvement before the program scales further.
Evaluation Methods
The evaluation followed Chyung's 10-Step Evaluation Procedure, which organizes the process into three phases: Identification, Planning, and Implementation. Patton's Utilization-Focused Evaluation approach guided the team's orientation throughout, keeping the intended users and uses of the findings front and center from the start.
Identification Phase: Evaluation Purpose and Type
Working with Berg and Gibson, the team established that the evaluation's primary purpose was to assess how well the NEO Program prepares new employees for their roles, with a focus on continuous improvement. This was a formative evaluation, designed to identify strengths and weaknesses and recommend adjustments before the program scales to all branches. It used a goal-based approach, measuring the program against its stated objectives, while remaining open to surfacing unintended outcomes.
The intended users of the findings were Berg and Gibson, who would use the results to guide revisions to program content, delivery, and design.
Planning Phase: Dimension and Data Collection
The team developed the evaluation's core dimension using the Kellogg Foundation's Program Logic Model (PLM), which maps program resources and activities to intended outputs, outcomes, and organizational impact. Working with stakeholders to confirm priorities, the team landed on a single dimension rated as "Very Important":
Program Design Effectiveness for New Employee Preparedness:
How well is the NEO Program (eLearning, live sessions, schedule, support, etc.) designed to help new employees feel prepared to start their roles at Tamrack?
Four data collection methods were selected to measure this dimension using a mixed-methods approach:
Anonymous web-based survey administered to Tamrack managers
Extant data review of NEO participant surveys from the past year
Extant data checklist review of the existing eLearning course
Extant data checklist review of Tamrack's intranet site
Methods were chosen to complement each other's strengths and limitations. Each was tied directly to the PLM and the evaluation dimension, with a scoring rubric developed for each instrument to allow for consistent analysis.
Implementation Phase: Develop, Collect, Analyze, and Conclude
During the Implementation Phase, the team developed the data collection instruments, gathered data, and analyzed findings.
The manager survey was developed as an anonymous, web-based questionnaire covering four areas: new-employee preparedness for the role, overall program effectiveness, areas for improvement, and open-ended questions. It was distributed by Berg to nine managers, with five completing it, a 56% return rate. Checklists for the eLearning course and intranet site were developed independently by the team, focusing on navigation, design, accessibility, and usability.
Data from all four sources were analyzed against established rubrics and then triangulated using a dimensional rubric that consolidated results across instruments. Three team members independently reviewed extant data to reduce the risk of biased interpretation. Open-ended responses from employee and manager surveys were coded and organized into a SWOT analysis to identify themes across the qualitative data.
The triangulated results across all four sources produced a single-dimensional score for Program Design Effectiveness for New Employee Preparedness. The combined findings rated the NEO Program as "Effective," supporting conclusions and recommendations delivered to Berg and Gibson in a final written report.
Results
Across all four data sources, the NEO Program scored within the "Effective" range on the evaluation rubric.
New-employee survey data from the past year showed 60-79% positive ratings. Manager survey results were stronger, falling within the "Highly Effective" range, with average scores between 4.0 and 5.0. The eLearning course review and the intranet usability checklist both fell within the "Effective" range.
A SWOT analysis of open-ended responses from employees and managers surfaced a more nuanced picture. Strengths included role-playing exercises, hands-on training, and thorough introductions to company culture and policies. Weaknesses centered on content overload, insufficient interactive elements, and unclear mentorship roles. Opportunities identified included scenario-based learning and a more structured mentorship program. Threats included the risk of reduced engagement due to poor pacing and gaps in technical support.
Conclusions
The evaluation confirmed that Tamrack's NEO Program is achieving its core purpose: new employees are leaving onboarding feeling reasonably prepared and welcomed. That's a meaningful baseline, particularly for a program that has been revised and refined entirely online since 2020.
The more important finding for Tamrack's next phase is where the program falls short. Content pacing, mentorship clarity, and interactive learning design are the areas most likely to affect employee readiness as the program scales to more locations and a growing employee base. The team's recommendations targeted these gaps directly, including tools like onboarding checklists and video tours, pacing adjustments, breaking up lengthy eLearning modules, and establishing clearer mentorship expectations.
The evaluation also surfaced a limitation worth noting: with access limited to two upstream stakeholders and a 56% response rate among managers, the findings represent a narrower view than is ideal. Expanding stakeholder access in future evaluation cycles would considerably strengthen the evidence base.
References
American Evaluation Association (AEA). (n.d.). Guiding principles for evaluators. https://www.eval.org/About/Guiding-Principles
Chyung, S. Y. (2019). 10-step evaluation for training and performance improvement. Sage.
Health and Human Services Department. (2006). Research-based web design & usability guidelines. https://www.hhs.gov/sites/default/files/research-based-web-design-and-usability-guidelines_book.pdf
Kellogg, W.K. (2004). W.K. Kellogg foundation logic model development guide. WK Kellogg Foundation. https://wkkf.issuelab.org/resource/logic-model-development-guide.html
Knowles, M.S. (1988). The modern practice of adult education: From pedagogy to andragogy. Cambridge Adult Education.
Mayer, R.E. (2017). Using multimedia for e-learning. Journal of Computer Assisted Learning, 33, 403-423. https://onlinelibrary.wiley.com/doi/am-pdf/10.1111/jcal.12197
Petrie, H., Papadofragkakis, G., Power, C., & Swallow, D. (2009). Navigational consistency in websites: What does it mean to users? Lecture notes in computer science, 423–427. https://doi.org/10.1007/978-3-642-03655-2_46
Sweller, J. (1988). Cognitive load during problem solving: Effects on learning. Cognitive Science, 12, 257-285. https://onlinelibrary.wiley.com/doi/epdf/10.1207/s15516709cog1202_4
TAMRACK. (n.d.). https://sites.google.com/boisestate.edu/tamrack/home?authuser=0
Yarbrough, D.B., Shula, L.M., Hopson, R.K., & Caruthers, F.A. (2010). The Program evaluation standards: A guide for evaluators and evaluation users (3rd. ed). Thousand Oaks, CA: Corwin Press. https://evaluationstandards.org/program/
Appendices
Note: Permission to reference the project sponsor and associated organizational group has been granted for the purposes of this portfolio. All information included reflects approved use and adheres to the confidentiality guidelines established for the project.