The third issue of the Journal of Expanded Learning Opportunities (JELO) has arrived! This spring issue launched at the 2016 BOOST Conference and features a conversation about quality programming in afterschool, an article on the role that social-emotional learning can play to close the achievement and learning gaps, and an article focusing on the links between professional development and quality STEM learning experiences. You can visit the first week’s installment about social emotional learning, last week’s piece with a researcher and practitioner conversation. Today’s blog features a study conducted by Deborah Lowe Vandell, Ph.D., Rahila Simzar, Ph.D., Pilar O’Cadiz, Ph.D. and Valerie Hall, Ph.D. from the University of California, Irvine.
Author Note
This project was supported by funding from the S. D. Bechtel Jr. Foundation, the Noyce Foundation, and the Samueli Foundation in collaboration with the Afterschool Division of the California Department of Education. The views expressed in this paper are those of the named authors and are not necessarily the views of the project funders.
Abstract
This study reports the results from a STEM learning initiative involving 96 public funded afterschool programs in California. Relations between professional development, staff beliefs, quality of STEM learning activities, and changes in student outcomes were examined over an academic year (2013-2014). STEM professional development experiences were linked positively to program staff beliefs about the value of STEM learning, which were linked to the quality of STEM learning activities reported at the programs, which were linked to several student outcomes, including gains in student work habits, math efficacy, science efficacy, and science interests. These findings support the utility of STEM professional development in afterschool settings.
Findings From an Afterschool STEM Learning Initiative: Links to Professional Development and Quality STEM Learning Experiences
Improving the quality of science, technology, engineering, and math (STEM) education has become a national priority (Dabney et al., 2012; Krishnamurthi, Ballard, & Noam, 2014; National Research Council, 2011, 2012; Simzar & Domina, 2014). Although the majority of these efforts during the K-12 period have focused on improving in-school STEM learning, there is a growing awareness of the potential role of afterschool programs in promoting STEM learning (Bell, Lewenstein, Shouse, & Feder, 2009; National Research Council, 2015). However, efforts to introduce ongoing and high quality STEM experiences in out-of-school (OST) settings face serious challenges. One challenge is that a substantial proportion of afterschool staff members have limited education and training in STEM subjects (Vandell & Lao, 2015). A second challenge is high staff turnover (Vandell & Lao, 2015). A third challenge is structural barriers—many afterschool programs have weak relationships with host schools, which limit programs’ access to STEM learning materials and opportunities to coordinate activities with classroom teachers (Bennett, 2015).
The purpose of the present study is to examine the effects of an afterschool professional development initiative in the State of California to determine (a) if professional development activities are linked to program staff beliefs about the importance of STEM learning; (b) if, in turn, staff beliefs are related positively to quality of STEM-related activities in the afterschool classrooms; and (c) finally, if the quality of STEM-related experiences is associated with changes in student STEM-related dispositions over an academic year.
A Compelling Need for Staff Professional Development
Although program staff are charged with leading engaging and meaningful learning activities at afterschool programs, their education and training is typically more limited than K-12 classroom teachers. K-12 classroom teachers have four-year college degrees as a minimum, and the majority (56%) have a master’s degree or more (U.S. Department of Education, 2010). In contrast, less than half of afterschool staff members have four-year degrees and less than 20% have a master’s degree (Nee, Howe, Schmidt, & Cole, 2006). In addition, K-12 classroom teachers complete hundreds of hours of pedagogical training and supervised field experiences prior to becoming the instructor of record in their classrooms. Staff members in afterschool programs do not typically undergo this type of preparation (Nee et al., 2006).
Thus, while many afterschool staff members bring energy and commitment to their work, there is a great need to expand staff development opportunities for further education and training in the field, especially if programs seek to expand their offerings to include enriched STEM (Dennehy & Noam, 2005). The present study examines the effects of one such effort to offer professional development at multiple afterschool sites. Here, professional development refers to a diverse set of activities such as trainings offered by other organizations, informal and formal meetings among staff members, meetings with classroom teachers, and coaching by internal and external advisors.
Context for the Present Study
There are a growing number of public and private efforts to create meaningful STEM learning opportunities in afterschool contexts (Bevan & Michalchik, 2013; Krishnamurthi et al., 2014). Included in these efforts is the work of 17 statewide afterschool networks that have sought to coordinate efforts to support afterschool STEM learning (National Research Council, 2015). The present study focuses on one such initiative that was developed by the California Afterschool Network and a consortium of foundations. This statewide initiative was a three-year project aimed at increasing STEM learning opportunities in publicly funded afterschool programs serving low-income, ethnically diverse students.
Figure 1 presents the logic model underlying this state-level initiative. The logic model is sequential, with Professional Development and Curricula Innovation support represented in the box on the left side of Figure 1.
Figure 1.
Figure 1. Logic model for the out-of-school time STEM initiative. Professional Development and Curricula Innovation support is represented by the box on the left. Professional development was expected to yield improvements in (a) Staff Beliefs about the value of STEM learning and feelings of efficacy when implementing STEM activities, and (b) Program Offerings (the quantity and quality of STEM activities offered by programs). Staff Beliefs and Program Offerings were expected to be mutually reinforcing, as illustrated by the bi-directional arrow between the two circles. Staff Beliefs and Program Offerings were then expected to yield improvements in Student Outcomes, the diamond box on the far right of the figure. Student outcomes included student reported work habits, student reports of efficacy in math and science, science interest, and career aspirations in the STEM domain.
Professional development was expected to yield improvements in (a) staff beliefs about the value of STEM learning and feelings of efficacy when implementing STEM activities, and (b) the quantity and quality of STEM activities offered by programs. Staff beliefs and program offerings were expected to be mutually reinforcing, as illustrated by the bi-directional arrows. Staff beliefs and program activities were then expected to yield improvements in student outcomes, the box on the far right of the figure. Student outcomes included student reported work habits, feelings of efficacy in math and science, science interest, and career aspirations in the STEM domain. These student dispositions are important predictors of students’ likelihood to pursue STEM topics in the future (Bell et al., 2009; Bevan & Michalchik, 2013).
Method
Participants
A total of 601 afterschool program sites, located in five of California’s afterschool regions, participated in the STEM learning initiative in 2013-14. These five regions were originally selected in 2012-13, following a statewide competition. As part of the initiative, programs received technical assistance from Regional Innovation Support Providers (RISPs) who facilitated access to high quality staff training materials and curricular resources and who assisted partnerships among programs and support agencies. In this paper, we focus on the effectiveness of the initiative in 2013-14 at 96 program sites with all five regions represented by at least eight program sites.
Measures
A research team from the University of California, Irvine was responsible for overseeing data collection. Surveys were administered to program staff and to students using an online format. Program staff also reported the quantity and quality of STEM activities on a daily basis using STEM Activity Documentation Forms.
Program staff surveys. Online surveys were designed based on studies and administered to 178 staff in fall of 2013 and to 90 staff in spring of 2014 in which program staff reported various demographic characteristics (gender, age, ethnicity), educational background (highest level obtained), professional experience, and job tenure in their current position (Noam & Sneider, 2010). Staff reported their professional development activities, which included how often they attended (1) general professional development training, (2) STEM-related trainings, (3) staff meetings on general topics, and (4) staff meetings on STEM topics in the past academic year. Staff also reported how often they met with classroom teachers to discuss STEM concepts being taught in school (Vandell, Warschauer, O’Cadiz, & Hall, 2008). A complete list of these measures and corresponding items are provided in Appendix A.
Staff reported their beliefs about the value of STEM learning for youth and their feelings of confidence (efficacy) when implementing STEM learning activities (adapted from Vandell et. al., 2008). Staff beliefs about the value of STEM learning for youth was assessed with seven items (e.g., “I think students look forward to coming to the afterschool program when we have STEM activities going on”). Staff efficacy for implementing STEM activities was assessed with seven items asking staff to report on their sense of competency leading STEM activities (e.g., “I feel confident about teaching Science, Technology, Engineering, and/or Mathematics in the afterschool program”). These constructs were scored on a 5-point Likert scale ranging from 1 (strongly disagree) to 5 (strongly agree). A complete list of items and internal consistencies of the scales for pre- and post- surveys, which were all acceptable, are provided in Appendix B.
Student surveys. Online surveys based on literature were administered to 3,738 students in fall 2013 and to 1,871 students in spring 2014. Students self-reported their work habits, math efficacy, science efficacy, social competencies, science interest, and science career aspirations (Noam & Sneider, 2010; Tyler-Wood, Knezek, & Christensen, 2010; Vandell, et al., 2008). Students’ work habits were assessed using six items (e.g., “I follow the rules in my classroom”). Both efficacy measures (math and science) were assessed using four items each (e.g., “I am good at math/science”). Science interest was assessed using 22 items (e.g., “Science is something I get excited about”). Social competencies were assessed using seven items (e.g., “I work well with other kids”) and students’ science career aspirations were assessed using four scales (e.g., “I will have a career in Science, Technology, Engineering, or Mathematics”). These constructs were scored on a 4-point Likert scale ranging from 1 (strongly disagree) to 4 (strongly agree). A complete list of items and internal consistencies of the scales for pre- and post- surveys ranging from acceptable to excellent, are provided in Appendix C.
STEM Activity Documentation Forms. These forms were developed by the authors to document specific activities at program sites. Staff recorded the following information about each STEM activity that was implemented: (a) date and duration of the activity; (b) number of students participating in the activity; (c) name of activity and STEM content area addressed; and (d) 4-point ratings of the level of student engagement, level of challenge, and overall assessment of success of the activity. A total of 2,457 STEM activities were reported during 2013-14.
Results
Program Staff
A total of 178 program staff at 78 sites reported their background characteristics. As shown in Table D1 (Appendix D includes Tables 1 through 8), a substantial majority of the staff was female (72%). The staff was ethnically diverse: 46% were Hispanic, 25% were white, 11% were Asian and 6% were African American. The staff was relatively young, with almost half (49%) being between 18 and 25 years, and 30% being between 26 and 35 years. The educational background of the staff varied widely. One-fourth reported having completed a four-year college degree, and 10% reported having post-graduate education. The remainder (65%) had less than a college degree, with the highest proportion (1/3) reporting “some college.”
Staff reported diverse professional experience. The majority (61%) of the program staff reported having experience working in an afterschool setting (e.g., leading activities and/or working directly with youth) and approximately half (51%) of the program staff had experience working as a classroom aide or teaching assistant. Finally, staff reported the length of employment at the program site. Here, 29% reported working at the respective program sites for less than six months. Almost half of the program staff (47%) reported having worked at their program site for less than three years.
Program Students
Surveys were completed by 3,738 students during the fall 2013 data collection. These students were fairly evenly divided by gender (49% male and 51% female). The majority of the students were in elementary school, with most of the students (72%) being in Grades 3 through 5. Twenty percent of the students who provided surveys were in middle school. Less than 1% of the students were in high school (Grades 9 through 12).
Types of STEM Activities That Occurred in the Afterschool Programs
A total of 2,457 STEM activities were reported by 84 staff at 53 program sites. As shown in Table D2, the majority (55%) of STEM activities focused on science. Typically, 28 students participated in each activity. Activities were between 30 and 59 minutes in duration. The majority of the reported activities involved students who were in third, fourth, and fifth grade (46%, 54%, and 47%, respectively). Staff reported that students were “mostly” engaged during 36% of the activities implemented and that they were “very” engaged during 56% of the activities implemented (an average of 3.48 on a rating scale from 1 to 4). Lastly, staff reported that the activities implemented went “mostly” well approximately 38% of the time and “very” well approximately 53% of the time (an average of 3.43 on a rating scale from 1 to 4).
Professional Development as it Relates to Staff Beliefs About STEM Learning
Our first substantive analysis asks if specific types of professional development were related to staff beliefs about the importance of STEM learning for youth and to staff feelings of efficacy when implementing STEM activities. Tables D3 and D4 present standardized regression coefficients predicting staff beliefs about the importance of STEM learning and efficacy for implementing STEM activities, respectively.
In Table D3, Models 1, 2, 3, and 4 examine associations between specific types of professional development activities and staff beliefs about the importance of STEM learning. Model 1 indicates that higher levels of staff training during the past academic year is associated with a .32σ increase in staff-reported beliefs about the importance of STEM learning. Model 2 indicates that a one-σ higher level of STEM staff attending training during the past academic year is associated with a .29σ increase in staff-reported beliefs about the importance of STEM learning. Models 3 indicates that a one-σ increase in the frequency of staff meetings to discuss program issues is associated with a .29σ increase in staff-reported beliefs about the importance of STEM learning. Lastly, Model 4 indicates that a one-σ increase in the frequency of staff meetings to discuss STEM programming is associated with a .27σ increase in staff-reported beliefs about the importance of STEM learning.
In Table D4, Models 1, 3, 4, 5, and 6 show associations between specific types of professional development activities and staff feelings of efficacy when implementing STEM activities. Model 1 indicates that, on average, a one-σ increase in staff attending training during the past academic year is associated with a .29σ increase in staff-reported efficacy for implementing STEM activities. Model 3 indicates that a one-σ increase in the frequency of staff meetings to discuss program issues is associated with a .30σ increase in staff-reported efficacy for implementing STEM activities. Model 4 indicates that a one-σ increase in the frequency of staff meetings to discuss STEM programming is associated with a .36σ increase in staff-reported efficacy for implementing STEM activities. Model 5 indicates that a one-σ increase in the frequency of staff meetings with classroom teachers to discuss STEM concepts being taught in school is associated with a .28σ increase in staff-reported efficacy for implementing STEM activities. Lastly, Model 6 indicates that a one-σ increase in the frequency of staff meetings with parents about STEM activities is associated with a .23σ increase in staff-reported efficacy for implementing STEM activities.
Staff Beliefs Linked to the Quality of STEM Learning Activities
Our second set of substantive analyses asks if staff beliefs are linked to the quality of STEM activities at the afterschool programs. Table D5 presents the standardized regression coefficients relating staff beliefs to two measures of STEM activity quality and Table 6 presents standardized regression coefficients relating staff efficacy for implementing STEM activities to two measures of STEM activity quality. The analytical model views activity quality as a product of these staff beliefs net of determinants such as staff gender, ethnicity, and the number of students participating in the activity. Because the reports of STEM activities reported by staffs that share a site are not independent, we clustered standard errors on site identification to account for the non-random assignment of staff into sites.
In Table D5, Models 1 and 2 indicate that, on average, a one-σ increase in staff beliefs about the importance of STEM learning is associated with a .25σ increase in staff reports of student engagement during STEM activities and a .14σ increase in staff reports of how well the STEM activities went overall. In Table D6, Models 1 and 2 indicate that a one-σ increase in staff efficacy for implementing STEM activities is associated with a .27σ increase in staff reports of student engagement during STEM activities and a .09σ increase in staff reports of how well the STEM activities went overall.
The Quality of the STEM Learning Activities Related to Student Outcomes
Our third set of analyses asks if the quality of the STEM learning activities predicts changes in student outcomes over the academic year. Tables D7 and D8 present standardized regression coefficients predicting six student outcomes (work habits, math efficacy, science efficacy, social competency, science interest, and science career aspirations). The analytical model views each student outcome as a function of prior functioning in the domain and other determinants such as measures of activity quality (student engagement and how activities went overall) and student gender. Because student outcomes for students that share a site are not independent of one another, we cluster standard errors on site identification to account for the non-random assignment of students into sites.
Student engagement in STEM activities. In Table D7, Models 1 through 5 show significant relations between staff reports of student engagement in STEM activities and student outcomes. Specifically, Model 1 indicates that, on average, a one-σ increase in staff reports of student engagement during STEM activities is associated with a .06σ increase in student reports of work habits. Models 2 and 3 indicate that a one-σ increase in staff reports of student engagement during STEM activities is associated with a .06σ increase in student reports of math efficacy and a .13σ increase in student reports of science efficacy, respectively. Model 4 indicates that a one-σ increase in staff reports of student engagement during STEM activities is associated with a .18σ increase in student reports of social competency and Model 5 indicates that a one-σ increase in staff reports of student engagement during STEM activities is associated with a .08σ increase in student reports of science interest.
Overall STEM activity quality. In Table D8, Models 1 through 5 show significant relations between staff reports of how well the STEM activities went overall and student outcomes. Specifically, Model 1 indicates that, on average, a one-σ increase in staff reports of how well the activities went overall is associated with a .08σ increase in student reports of work habits. Models 2 and 3 indicate that a one-σ increase in staff reports of how well the activities went overall is associated with a .14σ increase in student reports of math efficacy and a .04σ increase in student reports of science efficacy, respectively. Model 5 indicates that a one-σ increase in staff reports of how well the activities went overall is associated with a .20σ increase in student reports of social competency and Model 5 indicates that a one-σ increase in staff reports of how well the activities went overall is associated with a .11σ increase in student reports of science interest.
Discussion
This study examined relations between professional development, staff beliefs, program activities, and student outcomes in a large, systemic effort to support STEM learning in California afterschool programs. The logic model guiding the initiative posited that specific types of professional development activities would relate positively to staff beliefs about the value of STEM programming, which would relate to the quality of STEM activities offered at the afterschool programs, which were expected to support gains in student outcomes.
Findings were consistent with this theory of change. In particular, staff who were exposed to more training activities (both general and STEM-specific) and who attended more staff meetings to discuss general program issues and STEM programming reported stronger beliefs about the value of STEM learning and stronger feelings of efficacy when implementing STEM activities. These findings support the value of a multi-prong approach to professional development within the afterschool context, one that incorporates dedicated training activities, staff meetings, and close links with host schools (Vandell & Lao, 2015).
Also consistent with the STEM initiative’s theory of change, the current study found that these staff beliefs were linked to the quality of STEM activities at the participating programs. Staff who endorsed the importance of STEM learning and who felt capable of implementing STEM activities reported higher levels of student engagement in the afterschool programs’ STEM activities and the overall quality of the STEM activities implemented. Links between staff beliefs and their practices have been reported in the early childhood (Sheridan, Edwards, Marvin, & Knoche, 2009; Zaslow, 2009) and K-12 in-school (Loucks-Horsley, Stiles, Mundry, Love, & Hewson, 2010) contexts, but have not been specifically studied previously in afterschool programs.
Finally, student engagement in STEM activities in the afterschool programs predicted relative gains in students’ work habits, math efficacy, science efficacy, social competency, and science interest over the school year. The strongest relations were found between student engagement and students’ math efficacy and social competency. These findings represent one of the first cases in which STEM professional development has been linked to positive student outcomes in the afterschool context.
It is noteworthy that the program staff who participated in the current initiative are similar to the staff profile at many U.S. afterschool programs (National Research Council, 2015; Peter, 2002, 2009; Vandell & Lao, 2015). A substantial majority of the program staff in the current study had less than a college degree. The majority of the program staff members were young adults, between 18 and 25 years of age and had brief tenures in their current position. Almost one in three of the program staff reported working at the program for less than six months. Because their education, training, and prior experience is limited, staff may particularly benefit from ongoing and continuing professional development opportunities that provide curricula supports accompanied by dedicated trainings and opportunities to connect with other program staff, parents, and classroom teachers on STEM-related topics. Importantly, these experiences can enrich students’ STEM experiences in afterschool settings and support growth in students’ interests and efficacy in the STEM domain.
For breakfast, Deborah had a bowl of cereal topped with fresh peaches.
Pilar had his favorite Sunday brunch breakfast, which he makes all the time for my family – Mexican Huevos con Chilaquiles (scrambled eggs mixed with fried tortillas, veggies, chile salsa and cheese).
Rahilia enjoyed a piece of toast and scrambled eggs with cheese.
References
Bell, P., Lewenstein, B., Shouse, A. W., & Feder, M. A. (Eds.). (2009). Learning science in informal environments: People, places, and pursuits. Washington, DC: The National Academies Press.
Bennett, T. L. (2015). Examining levels of alignment between school and afterschool and associations with student academic achievement. Journal of Expanded Learning Opportunities, 1(2), 4-22.
Bevan, B., & Michalchik, V. (2013). Where it gets interesting: Competing models of STEM learning after school. Afterschool Matters, 17, 1-8.
Dabney, K. P., Tai, R. H., Almarode, J. T., Miller-Friedmann, J. L., Sonnert, G., Sadler, P. M., & Hazari, Z. (2012). Out-of-school time science activities and their association with career interest in STEM. International Journal of Science Education, Part B, 2(1), 63-79.
Dennehy, J., & Noam, G. (2005). Evidence for action: Strengthening after-school programs for all children and youth: The Massachusetts Out-of-School Time Workforce. Achieve Boston, An Initiative of Boston After School & Beyond.
Krishnamurthi, A., Ballard, M., & Noam, G. G. (2014). Examining the impact of afterschool STEM Programs. Afterschool Alliance.
Loucks-Horsley, S., Stiles, K. E., Mundry, S., Love, N., & Hewson, P.W. (2010). Designing professional development for teachers of science and mathematics (3rd ed.). Thousand Oaks, CA: Corwin.
National Research Council. (2011). Successful K-12 STEM education: Identifying effective approaches in science, technology, engineering, and mathematics. Committee on Highly Successful Science Programs for K-12 Science Education. Washington, DC: The National Academies Press.
National Research Council. (2012). A framework for K-12 science education: Practices,
crosscutting concepts, and core ideas. Committee on Conceptual Framework for the New K-12 Science Education Standards. Board on Science Education, Division of Behavioral and Social Sciences and Education. Washington, DC: The National Academies Press.
National Research Council. (2015). Identifying and supporting productive programs in out-of-school settings. Committee on Successful Out-of-School STEM Learning, Board on Science Education, Division of Behavioral and Social Science and Education. Washington, DC: The National Academies Press.
Nee, J., Howe, P., Schmidt, C., & Cole, P. (2006). Understanding the afterschool workforce:
Opportunities and challenges for an emerging profession. National Afterschool Association for Cornerstones for Kids.
Noam, G., & Sneider, C. (2010). Common goals, common assessments: Evaluation guidelines for youth programs. Unpublished manuscript, Program in Education, Afterschool and Resilience (PEAR). Harvard University, Cambridge, MA.
Peter, N. (2002). Outcomes and research in out-of-school time program design. Philadelphia, PA: Best Practices Institute.
Peter, N. (2009). Defining our terms: Professional development in out-of-school time. Afterschool Matters, 8(2), 34-41.
Sheridan, S. M., Edwards, C. P., Marvin, C. A., & Knoche, L. L. (2009). Professional development in early childhood programs: Process issues and research needs. Early Education and Development, 20(3), 377-401. doi: 10.1080/10409280802582795
Simzar, R., & Domina, T. (2014). Attending to student motivation through critical practice: A recommendation for improving accelerated mathematical learning. In S. Lawrence (Ed.), Critical practice in P-12 education: Transformative teaching and learning (pp. 66-116). Hershey, PA: Information Science Reference. doi:10.4018/978-1-4666-5059-6.ch004
Tyler-Wood, T., Knezek, G., & Christensen. R. (2010). Instruments for assessing interest in STEM content and careers. Journal of Technology and Teacher Education, 18(2), 341-363.
U.S. Department of Education, National Center for Education Statistics. (2010). Teacher attrition and mobility: Results from the 2008–09 teacher follow-up survey (NCES 2010-353).
Vandell, D. L. & Lao, J. (2015). Building and retaining a high quality professional staff for extended education. Manuscript submitted for publication.
Vandell, D. L., Warschauer, M., O’Cadiz, P., & Hall, V. (2008). Two year evaluation study of
The Tiger Woods Learning Center. Report to the Tiger Woods Foundation. Retrieved from https://faculty.sites.uci.edu/childcare/reports/#a2008
Zaslow, M. J. (2009) Strengthening the conceptualization of early childhood professional development initiatives and evaluations. Early Education and Development, 20(3), 527-536. doi: 10.1080/10409280902908833