TECAID’s External Evaluator, the University of Washington Center for Research & Evaluation for STEM Equity (UW CERSE), undertook ongoing evaluation of the TECAID project using quantitative and qualitative survey analysis, interviews and focus groups, and observation of TECAID activities.
At the beginning of the project, the TECAID PI Team and CERSE articulated four areas of expected outcomes: Individual, Team, Department, and Broader Community. These four areas informed CERSE’s evaluation questions.
CERSE conducted data collection for TECAID from 2014 to 2017, as shown below. Please note, the evaluation timeline here does not include observation of Virtual Learning Community meetings; those meetings occurred on a monthly basis.
Longitudinal Surveys (pre, during, and post design) were used to assess individual, departmental, and team changes in readiness, awareness, and actions taken to impact mechanical engineering department cultures in the areas of diversity, equity, and inclusion.
Department Chair Interviews were conducted in Year 1 of the TECAID project. These interviews enabled the evaluators to gather baseline qualitative data on the diversity and climate in each participating TECAID department to more fully assess the complex nature of departmental transformation.
TECAID Team Focus Groups were conducted after Workshop 4 to gain more understanding about how the TECAID experience impacted participants as individuals, teams, and departments.
Observation of TECAID Workshops 1, 2, 3, and 4 provided additional formative feedback to TECAID leaders. This process was guided by an observation protocol that highlighted active engagement of participants, perceived usefulness of knowledge, and skills learned.
Workshop Surveys 1, 2, 3, and 4 were fielded and analyzed. The survey results and reports from each workshop enabled the TECAID project team members to better understand each workshop’s impact and helped improve the project overall.
Observation of Monthly Virtual Learning Community calls offered insight on individual, department level, and broader awareness building—as well as inter-team learning dynamics.
Consulting about the TECAID Model took place in early 2017, after the TECAID workshops were completed. CERSE shared insights from TECAID’s formative and summative survey outcomes to help guide the formation of the TECAID Model for Preparing Engineering Faculty to Lead Department Change in Diversity, Equity, and Inclusion.
Team Accomplishment Summaries were created to document TECAID team progress, and analyze similarities and differences across projects.
A Summative Report of the impacts of the TECAID project on individuals, teams, and broader community levels will be compiled by December, 2017.