Data and Evaluation Toolkit

Recognizing the importance of using data to examine and improve program practice and outcomes, NCAN has produced or assembled a number of data- and evaluation-related resources for our members. Many, but not all, of these resources related to NCAN’s Common Measures, a member-developed and research-backed set of college access and success indicators. These resources are listed below with short descriptions. Please note that some of these resources are only available to NCAN members. Please send questions, concerns, or feedback to Bill DeBaun, Program Analyst, at debaunb@collegeaccess.org.

Common Measures Quick Reference
This short document lists all of the Common Measures and the abbreviated citations to the research related to them. NCAN’s Common Measures are separated into “essential” and “if available” indicators, as well as by “Access” and “Success” and subcategories therein. To view the full citations for each of the Common Measures, consult the Common Measures Handbook. (Last updated March 2016).

Common Measures Handbook (Members Only)
The Common Measures Handbook is a lengthy reference that examines each of the Common Measures indicators and suggests the form in which its related data are stored, data sources, technical and tracking notes, and related research. This document will evolve as new suggestions on tracking and collection are offered by NCAN members and as new research relevant to these measures enters the field. (Last updated March 2016).

Driving Toward Program Improvement: Principles and Practices for Getting Started with Data
The first in a two-part series from NCAN and Exponent PartnersIf you’re just getting started, dive into part one, Driving Toward Program Improvement: Principles and Practices for Getting Started With DataHere you’ll find best practices for building a data-driven mindset, refining your logic model, adopting data management practices, building your capacity, and more. 

Roadmap for Tracking Your Student Results: Program Data and Systems
The second in a two-part series from NCAN and Exponent Partners.  If you’re moving towards a more sophisticated approach, take a look at part two, Roadmap for Tracking Your Student Results: Program Data & Systems. You’ll get a review of some foundational steps, frameworks for data management with a system, analysis methods, and more. 

Data Usage and Platforms for College Access and Success: Insight from the Field
Curious about how other programs use data? Confused about which data platform might be a good fit for your organization? This white paper seeks to help you answer both of those questions. Written as a product of the Common Measures Learning Community, this paper includes data from a survey of NCAN members and profiles and testimonials of five different data platforms being used by college access and success programs nationally. Future editions of this white paper will consider even more data platforms in the field.

Program Evaluation Primer (Members Only)
The phrase “program evaluation” is often viewed suspiciously or defensively, but the process of program evaluation can be extremely helpful for understanding what a program is successful at, where it could improve, and where its activities can and cannot be measured. This primer, written in “FAQ” style, is for those trying to learn more about what a program evaluation is and isn’t and whether and when a program evaluation might be right for them.

NCAN Common Measures Program Dashboard Excel Tool (Members Only)
This Excel spreadsheet is an upgraded version of the data tool that was released by NCAN in 2013. In this version, programs to add student level data, and that data is aggregated to a program-level dashboard that shows the program’s current status on various NCAN Common Measures. For more information on this tool, consult the file’s Instructions tab. This tool will continue to evolve and be updated according to NCAN members’ needs and feedback. Although this tool is not a replacement for a full data platform, it may be helpful for programs getting their start with data.

Student Privacy Resources
The privacy and security of student data are both critically important. The U.S. Department of Education’s Privacy Technical Assistance Center (PTAC) is a wealth of information around practices and policies that help organizations stay compliant with the law and keep student data safe. The PTAC Toolkit is organized around five important topics: security best practices, data governance, data sharing/dissemination, legal references (FERPA and cross-agency), and disclosure avoidance. Beyond the toolkit, PTAC also operates a Help Desk that programs can contact with questions or concerns on student privacy.

StriveTogether also has a very handy guide to student privacy titled "Student Data Privacy Best Practices: Five Ways Community Organizations Can Ensure Effective and Responsible Data Use." If your organization uses student data, this is an important refresher in maintaining that data's security.

A Practical Approach to Program Data Webinar Series (Members Only) 
The collection, management, analysis, and reporting of data all affect a program’s capacity to serve its students, but these topics can sometimes be confusing. Even programs with a seemingly good handle on working with data can learn new tricks and techniques to help them get better.

In order to help college access and success programs improve their capacity with data, NCAN teamed up with Idealware to offer an exciting benefit for our members! From September 2015 to January 2016 we held a seven webinar series titled “A Practical Approach to Program Data.”

Topics progress from which questions to ask of data all the way through how to make use of data both internally and externally. Although programs will get the most benefit from viewing all seven webinars, each individual webinar is intended to stand alone.

Other Data-Related Resources

  • Logic models are important tools that organizations can use to show how the elements of its program—its resources, activities, outcomes, etc. —come together. Creating a logic model may be a required part of program evaluation, but it’s also a great way to simply explain your program to prospective donors. This Logic Model Development Guide from the W.K. Kellogg Foundation is an excellent, detailed resource for logic model newbies and devotees alike.
  • “Data sharing agreement” is a phrase that is likely to cause a wide range of emotions in program staff; apprehension, bewilderment, and anxiousness are just some of them. The process of setting up an agreement between two parties, be they schools, districts, or non-profit entities is one that can be confusing and complicated. Fortunately, the William T. Grant Foundation’s Research-Practice Partnerships series has an entire module devoted to Developing Data Sharing Agreements, which includes guiding questions, work samples, and other related documents. The rest of the series is also worth a look, but this is the module with the most utility for NCAN members. 
  • StriveTogether recently released a very handy Data Sharing Playbook that dovetails nicely with the above resource. The playbook is intended to “help community organizations effectively partner with schools on data-driven ways to improve education outcomes. This resource includes seven principles about how to begin and grow a data-driven initiative, as well as practical resources to help communities implement complex data partnerships with schools and other community partners.”
  • Organizations talk all the time about how they are data-driven, or at least want to be. But organizations do not become data-driven overnight. It’s a gradual process, but where does that process start? “Getting Started with Data-Driven Decision Making: A Workbook,” from the Nonprofit Technology Network and Idealware, asks just this question. This document is free (just put in your contact info) and well worth your time if your organization is search for square one to becoming data-driven. This guide goes step-by-step to identify questions your organization wants answered, the metrics that could answer them, who would use those metrics, how they would be collected and tracked, and more. This would make for both a great staff retreat activity and a reminder of all the steps to consider when expanding data tracking.
  • What are the specific skills to operate in the data science environment that finds itself with so much available data? The team at Leada published “The Data Analytics Handbook” to answer some of these questions. There are four volumes: data analysts and data scientists, CEOs and managers, researchers and academics, and “big data edition.” Using interviews from professionals involved in data-using top-tier organizations, the handbook attempts to get at what each of these groups should know, and ask, to get the most out of the information they have.
  • The Center for Education Policy Research at Harvard University hosts the Strategic Data Project, which pairs data analysts with school districts to perform high-impact data analyses. The SDP makes available its Toolkit for Effective Data Use. The toolkit is “a resource guide for education agency analysts who collect and analyze data on student achievement. Completing the toolkit produces a set of basic, yet essential, human capital and college-going analyses that every education agency should have as a foundation to inform strategic management and policy decisions.” Although these tools are aimed at those working in school districts, many of the skills will apply to work done in college access and success organizations. The toolkit includes dummy data sets and step-by-step directions for data cleaning using Stata software. This resource is probably not a light undertaking, but it could give a fledgling analyst some more experience with key data skills. Also of note from the SDP is the “Strategic Use of Data Rubric,” which can be used “as a basis for gathering evidence of data use across the organization allows educational leaders to identify specific areas for improvement and highlight specific steps to move the organization toward using data more strategically.”
  • If you work for a school district or your program works closely with a school district and you wish you could get a better system of metrics established to see how students, schools, and the district overall are performing, take a look at the College Readiness Indicator Systems (CRIS) Resource Series. This will take a little time to read through, but it’s worth it. Launched by the Bill & Melinda Gates Foundation, CRIS worked with research and school partners across the country to develop indicator systems around three dimensions of college readiness: academic preparedness, academic tenacity, and college knowledge. The resource series has six parts, but the chart in the link above describes what it included in each part and who is most likely to benefit from reading it.