Florida's 21st Century Community Learning Centers
On Site Evaluation Visits - Program Evaluation Unit
Announced Visits
The on-site evaluation method includes: (1) an in-depth interview with program administrators and other interested parties; (2) visits of 21st CCLC sites within the specified program, typically lasting between 30 minutes and 2 hours; and (3) a debriefing session where immediate feedback of strengths and challenges is provided. Prior to engaging in this process, a thorough review of available documentation is completed by the Program Evaluation Unit, to include a review of the approved grant application and amendments.
FOCUS OF INTERVIEW:
As with previous years, the evaluation visit will consist of an interview, site visits, and a debriefing. In addition to other topics, the interview content will include an evaluation of the following: data collection, integrity, analysis, and use (see descriptions below); evaluation of program objectives; activities to progress towards and meet program objectives; parental involvement and communication with parents; staff trainings and meetings; volunteers; services for students with special needs; service learning; safety and emergency procedures; sustainability (see expectations regarding sustainability below); and program-wide and site-specific accomplishments and challenges. As part of the evaluation interview, 21st CCLC programs are required to provide evidence (e.g., data, analyses, etc.) of the following elements related to your internal and external evaluation:
Data Collection: The program must provide evidence that data related to each objective are being appropriately collected (e.g., presenting the actual data collected on each objective to evaluators). Specific indicators of this category include quality of data collection plans, implementation of baseline data collection procedures, understanding of data collection procedures/plans by program and site staff, reliability and accuracy of data collection, contingency plans for data collection, and corrective adjustments made to data collection plans/procedures.
Data Integrity: The program must provide evidence that collected data are secured from unauthorized observation and use by such measures as password protection and adherence to administrative procedures such as those required by the local school board and/or the Health Insurance Portability and Accountability Act (HIPAA).
Data Analysis: The program must provide evidence that data related to each individual objective are appropriately collected and reviewed on a regularly scheduled basis. Evidence of adherence to this standard should include such evidence as a written schedule for data collection, written analytical procedures, actual formative reports from data analysis, as well as evidence that whoever is assigned the task of data analysis is qualified to execute these responsibilities.
Data Use: The program must provide evidence that data related to each objective is being appropriately disseminated and applied to improve program performance and services to students and their families. Objective evidence of fulfilling this standard includes such evidence as a written dissemination plan, actual evidence of positive data-initiated program change, and confirmation from site coordinators and other stakeholders that they are actually receiving data and appropriate analysis.
SUSTAINABILITY EVIDENCE:
As part of the 2008-2009 evaluation process and in keeping with the requirements of Florida’s Auditor General, the FDOE is requiring that all 21st CCLC sub-grantees provide written evidence of in-kind services/donations and partnerships proposed in the approved grant application (as identified in the budget narrative, sustainability plan, summary of partners, and/or partnership letters). Each program must provide documentation about its progress with respect to the following:
1) Documentation of how the level and value of in-kind service/contribution were determined when writing the grant application.
2) Documentation of progress towards obtaining these proposed in-kind services. Either:
a) Documentation that the program is progressing towards receiving the full in-kind services / funding proposed; or
b) Documentation that the program has fully obained the services/funding (e.g., a cashed check, a contract, etc.)
All documentation must be copied and given to the evaluators in HARD COPY form. These documents will be provided to the FDOE and placed in the permanent program files for review by the Auditor General.
STUDENT ATTENDANCE:
As part of the 2008-2009 evaluation process and in keeping with the requirements of the Department of Financial Services, all 21st CCLC sub-grantees must provide evaluators copies the last 10 days of actual attendance rosters for EACH SITE at the beginning of the evaluation visit. This is not simply numbers or counts of students, but is the full attendance rosters with student names for all sites in the program. This must be provided as a hard copy of the rosters, as it will be provided to the FDOE for placement in the permanent program files for review by the DFS and Auditor General. In addition to this requirement, all programs (even those not on performance based reimbursement) should to complete the Performance Standards Monthly Report (available at http://www.fldoe.org/curriculum/21century/21cclcrfp.asp), as it will likely be required of all programs in future years.
SITE VISITS:
Information regarding expectations of the site visits is included in the attached "Site Worksheet." While the selected sites to be visited are generally not announced and some may not be visited, the program must provide a completed "Site Worksheet" for each site in the program by the end of the evaluation visit.
DEBRIEFING:
The debriefing will take place following the interview and site visits. During the debriefing, the program evaluators will provide immediate feedback regarding program strengths and challenges.
In addition to the debriefing, the program will be provided a program-wide written report. Programs are strongly encouraged to take detailed notes during the debriefing process, as the written report may not include the same level of detail presented during debriefing sessions. The report will, however, outline the evaluators' findings with the intent of assisting you and your staff in sharing successes and addressing concerns. Reports will be distributed to the program director and grant administrators via email. It is anticipated that reports will be released within two weeks following the evaluation visit.
Unannounced Visits
The method for unannounced on-site evaluations includes: (1) interviews with site-level and school-level administrators present during the visit; (2) interviews with 21st CCLC program participants present during the visit; and (3) observation of 21st CCLC programming, typically lasting between 30 minutes and 2 hours. Prior to engaging in this process, a thorough review of available documentation is completed by the Program Evaluation Unit, to include a review of the approved grant application and amendments. Topics generally discussed during evaluation visits include, but are not limited to, the following: (1) program objectives and activities, (2) efforts towards data collection and assessment of program objectives; (3) parental involvement activities and communication with parents; (4) dropout prevention activities; (5) services provided to students with special needs (including average daily attendance for students with special needs); (6) safety and emergency procedures; (7) program-wide and site-specific accomplishments and challenge.