21st Century Community Learning Center (21st CCLC) grants represent the second largest source of funding for out-of-school time (OST) programs in the City of Philadelphia. Thirty-three OST providers in Philadelphia, representing more than 100 OST programs, are funded by these federal grants, which support OST programs that provide academic support for youth attending high-poverty, underperforming schools.
For the past three years, Research for Action (RFA) has served as the evaluator for multiple 21st CCLC grantees, and, in this role, has been able to aggregate evaluation findings to produce two reports on the citywide impact of 21st CCLC programs. RFA produced these reports in collaboration with the city of Philadelphia’s OST system-building efforts; the reports also fill a critical gap in information regarding the impact of OST programs in the city.
Key Findings: 2013-14
RFA released its second citywide 21st CCLC report in April 2014. For two years in a row, this research has documented positive relationships between student participation in 21st CLCC programs and academic and behavioral outcomes, particularly for students who are active participants in the programs. The research has also begun to explore common strengths and challenges for Philadelphia’s 21st CCLC programs.
For the 2013-14 school year, RFA conducted a mixed-methods evaluation of 11 grantees representing 50 programs that examined student demographics, elements of program quality and important student outcomes. The analyses in both years examined the potential impact of OST programs on school attendance; academic outcomes, including standardized test scores; and behavioral outcomes (Durlack & Weissberg, 2012; Harvard Family Research Project, 2008). After taking into account the preexisting differences in academic and behavioral performance and observed background characteristics, our year two analyses suggest:
- Regular elementary school OST participants were less likely to have 10 or more unexcused absences than were students who did not participate or who participated in less than 30 days of programming.
- Regular middle school OST participants were more likely to earn higher course grades in math and reading than were students who did not participate or who participated in less than 30 days of programming.
Furthermore, and similar to findings in the year one analysis, higher levels of participation were associated with better outcomes in many (but not all) behavioral and academic performance indicators. More specifically:
For elementary participants, high levels of participation are associated with:
- A lower chance of having 10 or more unexcused absences or out-of-school suspensions
- Higher reading course grades
- Higher Pennsylvania System of School Assessment reading and math scores
- A greater chance of reading at grade level for 1st through 3rd grade students
For middle school participants, high levels of participation are associated with:
- A lower chance of having 10 or more unexcused absences or out-of-school suspensions
- Higher reading, math and science course grades
- Higher PSSA reading scores
For high school participants, high levels of participation are associated with:
- A lower chance of having 10 or more unexcused absences or out-of-school suspensions
- A higher chance of earning all of the credits attempted in English and language arts and math
RFA’s evaluation also identified common strengths and challenges in program implementation.
First, analyses of participation data show that elementary school students were more likely to participate regularly than were middle and high school students. The majority of elementary school students participated in 21st CCLC programs at a meaningful level (90+ days, or half of the school year), but the majority of middle and high school students did not do so.
Program directors and staff understood that academic improvements were a clear goal for 21st CCLC-funded programs. To support these goals, programs generally offered robust academic supports through homework help, project-based learning (PBL) and academic enrichment activities. Programs supplemented academic support with enrichment activities such as sports or arts. Two challenges encountered by many programs were providing a significant amount of one-on-one or small group academic support and aligning the academic supports to school day activities, practices identified in other research as important to improving academic outcomes (Abt Associates, 2001; Black et al., 2009; Lauer et al., 2006).
Academic supports also included STEM (science, technology, engineering and math) programming, the focus of a subset of programs within the cohort of providers. Six providers offered STEM programming, including robotics, urban gardening and forensic science activities. However, these grantees, who were only in their first year of implementing these supports, experienced a number of challenges in providing opportunities for students to engage in hands-on, inquiry-based learning opportunities.
Overall, school partnerships, a foundational component of any academically focused OST program, were described as supportive. The 16 principals RFA interviewed all embraced the 21st CCLC programs that had goals aligned directly or indirectly with their own. While provider and principal reports of school partnerships were generally positive, there was significant variation among providers in the depth of the partnerships they developed with schools, particularly the degree to which they developed relationships with the classroom teachers of the students in their programs.
One strategy used by eight of 11 providers to build school partnerships was the utilization of overlapping or blended staffing models to ensure continuity between the school day and the OST program. Overlapping staffing is a model that ensures that OST staff are in the school building during part of the school day. Blended staffing is a model in which OST programs hire teachers or para-professionals from the host school to work in the OST program. These approaches created more opportunities for the OST programs to develop relationships with classroom teachers and other school staff. These models have been cited as promising practices in other literature on OST-school partnerships (Harvard Family Research Project, 2010).
Finally, providers demonstrated both promising practices and challenges with OST staffing, another foundational element of any OST program. Providers tended to employ staff with at least some college education, a characteristic that has been associated with more positive academic outcomes (National Center for Research on Evaluation, Standards, & Student Testing, 2011; RAND Corporation, 2005), but overall, OST staff were relatively new to working for their providers, which suggested that programs had experienced staff turnover. Staff reported having the greatest competency in developing relationships with youth and supporting homework help. Professional development was readily available to staff, but staff believed they could use additional support in student behavior management and parent and community communications.
RFA tested whether any of these program implementation factors were related to student outcomes. Results of these analyses were inconclusive but did point to some academic benefits for high school students when OST staff had higher levels of education or were certified teachers.
Recommendations
RFA’s findings suggest several recommendations for Philadelphia’s OST system and its providers.
First, OST providers should focus on increasing the levels of program participation, particularly for middle and high school students. In addition, programs serving high school students may consider ways to increase the percentage of program staff with bachelor’s degrees who work in their programs. Providers may also want to consider blended or overlapping staffing to facilitate the deepening of school partnerships.
At the citywide level, the OST system should continue to support citywide professional development of OST staff in behavior management, STEM programming and developing community resources and community connections. The School District of Philadelphia could also provide support for deepening OST-school partnerships including promoting more opportunities for teachers and OST staff to communicate about the needs of their students and align their supports.
RFA is continuing its evaluation work in the 2013-14 and 2014-15 school years and plans to release two additional citywide reports in subsequent years.