The full charter school advantage: What the CREDO reports overlook

I have written a number of times such as here, here, here, here and even way back when here in our old blog about what I think is the most important finding in an ever growing series of reports on charter schools from The Center for Research on Education Outcomes (CREDO) at Stanford University.

This important finding: it takes time to move students ahead when they usually enter charter schools with academic performance well behind their peers in the traditional public school system. CREDO has shown repeatedly – even in their very first report from 2009 – that once students spend several years in charter schools, they start to move dramatically ahead of their traditional public school counterparts.

But, CREDO’s research has an important short-coming related to this charters-need-time finding. CREDO shows charter school performance for students who have been in their charter for a year or more, but CREDO doesn’t show how students perform when they first enter a charter.

So, I decided to take a look at data in one of CREDO’s most positive charter school reports, their “Charter School Performance in Louisiana, 8/8/2013.” In its Louisiana report CREDO shows in Figure 9, “Impact by Students’ Years of Enrollment,” that by the time Louisiana students have spent four to five years in a charter, they are about 180 days ahead of their traditional public school counterparts. That is basically a full school year ahead.

But, how were these charter school students performing upon charter entry? CREDO’s graph doesn’t tell us.

So, I extrapolated the CREDO data for years one and two backwards to develop a picture of the charter entry situation. The figure below shows how that worked out.

extrapolation-of-credo-impacts-over-time-in-louisiana-2013-report-to-year-zero

In this figure, the numbers shown for Year 1, Year 2, Year 3 and Year 4 to 5 are derived from data in CREDO’s Figure 9 using a conversion to “days of learning” discussed in the report (for techies, see more on that at the end of the blog).The zero horizontal axis in the figure above represents the average performance found in Louisiana’s traditional public schools. So, for example, CREDO shows that at the end of their first year, Louisiana charter students are still about three school days behind in reading and about 14 days ahead in math.

CREDO also shows charter student performance further outpaces the traditional school students as charter students spend more time in Louisiana’s charter system. After students have been in charters from four to five years, they are about 180 days of learning ahead in math and 187 days of learning ahead in reading compared to their traditional public school counterparts. That’s about a full year of extra learning in both subjects.

But, how about what looks like fairly unimpressive performance after the first year in Louisiana's charters?

Extrapolating the CREDO data backwards to a charter school entry situation, I estimate that the average charter student in Louisiana enters these schools of choice about 58 days behind in math and 85 days behind in reading.

So, charter school students actually start to make notable improvement even in their first year of attendance. But, you can’t see this in CREDO’s reports because they don’t talk about student performance upon charter entry.

Thus, it looks like Louisiana charter students make notable progress even in their first year in charters. It’s something I hope CREDO investigates in more detail in the future. Charter schools don’t get a fair evaluation, otherwise.

There are important implications in this information.

For one thing, any charter school study that lumps all charter students together, including first year students, is going to under-report the real charter school advantage.

Failure to consider charter impacts over time explains why a number of charter reports don’t show charters performing very well. Very simply, by not allowing charters enough time to work with incoming students, those reports have a negative bias against charter schools.

This is a tremendously important issue, which undermines the accuracy of many charter studies. Now, you understand why.

Technical information:

While Figure 9 in CREDO’s report does have scales for both scoring in “standard deviations” and in “days of learning,” the more accurate numbers shown in the figure are in the more difficult to appreciate standard deviation scores.

To construct the figure in this blog, I used a conversion factor from standard deviations to days of learning that CREDO discusses in Table 3 of their report. That table shows a growth of 0.25 standard deviations is equivalent to 180 days of extra learning. That works out to 7.2 extra days of learning for each 0.01 standard deviation of growth, or 72 extra days of learning for a 0.1 standard deviation.

As CREDO points out, these conversions are approximations, but they do provide ball park estimates of considerable value for explaining the relative importance of the more obscure standard deviation scores.