At their finest, out-of-school time (OST) programs are vibrant environments that nurture and promote creativity and innovation, in the activities and opportunities offered as well as in the personal connections and social environments that lead to positive effects for children and youth. Too often, these moments of brilliance are just that—short-term instances made possible by individual staff who move on to other jobs without institutionalizing what worked, leaving programs to scramble to sustain consistent funding to support staff time for both work with youth and administrative functions.
National research demonstrates the potential for OST to contribute to improved behaviors and skills for youth, addressing crucial factors for staying on track to high school graduation such as classroom attendance, behavior and grades (Durlak, Weissburg, & Pachan, 2010; Kataoka & Vandell, 2013). Programs that include quality elements like caring staff and engaging activities that explicitly build skills and positive relationships with participants and families contribute most notably to positive impacts for children and youth who participate consistently over time. (Durlak, Weissburg, & Pachan, 2010; National Institute on Out-of-School Time, 2005). However, most OST programs rely primarily on low-salary part-time staff to facilitate daily program activities, with planning and management support from a smaller number of full-time staff. This and other resource limitations pose significant challenges for OST programs to sustain innovative approaches and the effective practices and quality elements that will best help children and youth achieve positive education and development outcomes.
Program quality assessment and improvement efforts informed by data on short-term youth outcomes can be vital tools to help programs focus their efforts on making improvements that lead to increased impact and better prepare youth for a successful transition to adulthood. In Philadelphia and across the state, nonprofit and system leaders are working together to expand available supports for programs to work internally and collectively to increase their quality and focus on targeted youth outcomes like positive relationships and academic engagement that contribute to longer-term achievement.
The Pennsylvania Statewide Afterschool/Youth Development Network (PSAYDN) formed in 2004 with the goal of promoting high-quality youth development programs across the state. PSAYDN and its members’ collective efforts have focused on increasing awareness of the value and elements of quality programs, expanding supports for quality improvement and advancing related system-building in different localities statewide. Working with and informed by the Keystone Stars quality improvement system for regulated child care serving children up to age 13, PSAYDN has concentrated on extending supports to include all program types and older youth offerings as well. These efforts over the past decade reflect the complexity and gradual process of institutionalizing effective innovations.
To first build knowledge and understanding among providers and stakeholders, PSAYDN created resource materials that synthesized common elements of program quality found in national research and in tools from other states. With a broad range of member input, PSAYDN developed and disseminated the Statement of Quality in Afterschool in 2006 as a concise reference, then compiled the accompanying PSAYDN Quality Self-Assessment and Discussion Guide in 2010 to facilitate discussion, reflection and improvement planning related to key areas of program quality. All of these resources provide information to support quality practices in how programs are designed, staffed and implemented.
Across Pennsylvania, youth programs receive funding from a variety of sources, with intermediary organizations or networks often providing support and monitoring program quality elements like qualified staff and program activities. Staff training is the most commonly available resource to support programs, followed by assessments of compliance or quality and, less often, technical assistance of some form. Although youth programs face increasing pressure to demonstrate impact, as does the entire nonprofit sector, few can manageably measure outcomes information or use that data to inform program improvements. PSAYDN’s initial work indirectly supported these efforts by increasing wide-scale understanding, with subsequent work moving to fostering the skills and support systems for program quality and continuous improvement.
In 2012, United Way of Greater Philadelphia and Southern New Jersey (UWGPSNJ) proposed a coordinated statewide planning process that would extend the local work of PSAYDN members to advance quality improvement systems (QIS). Focused initially on older youth and programs serving K-8 youth who were not in the Keystone Stars system, this effort sought to build internal program capacity to self-assess quality and to incorporate the use of outcomes data to inform program improvements. Ultimately, the addition of outcomes data collection expanded the relevance of the project to also include school-age care programs that were already part of Keystone Stars. Leveraging funding from UWGPSNJ and the Wallace Foundation, phase I of a QIS pilot began in 2013 with support from a statewide steering committee of PSAYDN members and with implementation by Philadelphia youth systems leaders and providers across various networks. Informed by their efforts, and with continuing support from the National Institute for Out of School Time (NIOST), PSAYDN is currently engaging with additional networks across the state to launch Phase II of the pilot starting in Fall 2014.
In Philadelphia, the BOOST system of systems collaborative coordinated local implementation of a quality improvement and outcomes pilot during the 2013-14 school year. A leadership team with representatives from the Department of Human Services (DHS), PHMC, the Free Library and the Department of Recreation has worked closely with a contracted project coordinator to identify participating programs and to manage training and technical assistance supports for data collection and quality improvement. PHMC and DHS implemented a formalized continuous quality improvement process in their OST network , which entailed nearly all sites completing the PSAYDN quality self-assessment and 18 programs instead using NIOST’s APAS (Afterschool Program Assessment System) pilot and APT-O (Assessment of Program Practices Tool for Observation) assessment and also incorporating data collection and analysis using NIOST’s Survey of AfterSchool Youth Outcomes (SAYO). Using the results from these initial assessment data, PHMC program specialists worked with programs to define two or three priority quality improvement goals and related action steps to be achieved by the end of the school year. Program Specialists followed up with sites throughout the school year to monitor and support their progress. This process will continue in 2014-15 with more programs beginning to collect outcomes data and incorporate that pre/post data into quality improvement planning.
Moving forward in other regions and funding networks without the benefit of available funding to expand a statewide system, PSAYDN is taking a multilevel approach—supporting networks’ existing efforts while providing coordination to define and advance quality system-building goals across the state. The QIS pilot and work to date have helped distill insights into what is needed to institutionalize a culture of continuous quality improvement and more effectively support measurable impact on youth outcomes. Recruiting new networks and programs to take part in quality improvement, as well as securing resources to support this work, requires a compelling case for the benefits (What’s in it for me?) and a clear understanding of expectations (What’s required of me?). Connecting quality improvement efforts to increased resources and knowing program efforts are making a difference are key motivators. Personal relationships are essential for securing buy-in, working together towards shared goals and making the culture change needed to institute quality improvement. Determining some common youth outcomes to measure across networks—starting with positive relationships and engagement in learning—sets the stage for collective impact, supported by vetted assessment tools and QIS core components. All participating networks and programs will incorporate quality self-assessment, measure targeted short-term outcomes and support work toward quality improvement plan goals. Across Pennsylvania, quality improvement efforts might be tied to a particular funding stream or a more grassroots approach like the provider-driven Quality Campaign of the Allegheny Partners for Out of School Time. Options for variations within networks are offered within the agreed-upon framework. By building networks’ capacities to support outcomes data collection and quality improvement, PSAYDN partners strengthen the field and help increase public support, increasing the number of programs that demonstrate and sustain impact and ultimately leading to more positive outcomes for the children and youth who participate.