The Prichard Committee’s Blog weighed in on March 11, 2012 about Kentucky’s nation-leading levels of exclusion of students with learning disabilities in the National Assessment of Educational Progress (NAEP) reading assessments.

Sadly, Prichard’s blog glossed over the exclusion problem, claiming, “Kentucky's recent record of relative NAEP success does not, in fact, evaporate when exclusion rates are considered.”

Well, that’s just not so. In Part 1 I discussed some very important testing policies, which Prichard ignored, that led to Kentucky’s NAEP exclusion problem. Now, I’ll discuss technical deficiencies in Prichard’s analysis of NAEP scores for those students who do not have learning disabilities.

To begin, I need to point out, again, that you have to look at NAEP data in the right way. You cannot get a fair picture of what is happening with state education programs using simplistic state to state comparisons only of “all student” NAEP test scores. The reason is student demographics now vary dramatically from state to state. In many other states high minority populations drag down the overall state average scores. Thanks to shifting racial demographics, it is actually possible for a state to make progress for each of its student racial groups but still fail to improve its overall average “all student” scores. There is even a name for this phenomenon: Simpson’s Paradox.

Because of this well-known problem, the most recent guidance from the NAEP 2011 Reading Report Card (See page 24) says:

“Differences in states’ demographic makeup should be taken into consideration when interpreting state results” (emphasis added).Prichard’s simplistic analysis doesn’t do that.

Take a look at what happens as soon as we disaggregate the data by race to see how non-learning disabled white fourth grade students in Kentucky match up to the national average reading scores for their counterparts in other states.

Kentucky’s white, non-learning disabled fourth grade students score statistically significantly LOWER than the national average today, as they did over a decade ago.

That is exactly opposite of what the “all student” analysis used by Prichard shows.

Proving the wisdom in the NAEP guidance about doing state to state comparisons, Prichard’s assertions regarding Kentucky’s reading performance for students who are not learning disabled start to fall apart as soon as the disaggregated data is examined. It isn’t appropriate to only conduct a simplistic analysis and then claim Kentucky’s reading performance is better than the national averages. In fact, Kentucky’s reading performance for the state’s dominant population of students is worse than elsewhere.

I must stress the fact that the need to disaggregate NAEP data when conducting state to state education comparisons is far from just my personal observation. It isn’t a new issue, either. NAEP report cards have pointed out the need to break out the data by race for over half a decade. I mentioned the most recent guidance from the 2011 NAEP earlier, but cautions in this area go back to at least the 2005 NAEP Reading Report Card. That report says on page 14:

“In comparing states to one another, it is important to consider that overall averages do not take into account the different demographics of the states’ student populations.”

The demographics issue is particularly important for any analysis of Kentucky’s performance. In sharp contrast to virtually every other state in the county, Kentucky’s schools today remain remarkably ‘white.’ Whites comprised 84 percent of the Kentucky’s public school student body as of 2011 NAEP testing. That gives Kentucky a huge, and unearned, advantage whenever simplistic “all student” test scores are compared across states where the national fourth grade average white enrollment is now down to 52 percent and runs as low as 25 percent in some states.

In fact, the NAEP 2009 Science Report Card actually uses Kentucky as an example of how things change radically when NAEP data is disaggregated by race. Here is the un-retouched Figure 32 from that report card, which shows Kentucky’s supposed better-than-national-average eighth grade science performance does not hold up when white student scores are examined.

This shows that the flaw in Prichard’s simplistic analysis of the reading scores can be found in other NAEP subjects, as well. And, even the people who create the NAEP know it.

Getting back to NAEP reading, I can hear the “But, we’re poor” whining starting, so here is a similar graph to the first one in this blog, but this one only shows results for Kentucky’s white fourth grade students who are not learning disabled but who are eligible for the federal free and reduced cost lunch program, a poverty measure.

Once again, the trend has been for Kentucky’s dominant student racial group, whites, to fall farther behind on NAEP fourth grade reading. As of 2011 we are definitely BEHIND, not ahead, of the national average for elementary school NAEP reading for poor, non-learning disabled whites in our fourth grade classrooms.

Things look a bit better, but not much, in the eighth grade results.

We are behind – again – by a statistically significant amount, but we did close the gap a bit.

Things look somewhat better still if we only look at Kentucky’s lunch-eligible eighth grade students without learning disabilities.

However, not even this last case supports Prichard’s claim that Kentucky’s non-learning disabled students score ahead of the rest of the nation for NAEP reading. Also, this graph shows Kentucky’s performance for poor eighth grade whites who are in the school lunch program is not improving compared to the rest of the nation. It has remained static for the past 13 years.

And, if all we want to do is compare ourselves to poor kids around other parts of the country, I submit we are not setting a suitable goal for our students.

Here are some other things Prichard’s blog left out.

Prichard didn’t tell you how Kentucky’s way above average exclusion rates on the NAEP now create enormous doubt about the validity of our NAEP reading scores.

Prichard also didn’t tell you that doubts about Kentucky’s real reading performance are especially strong when our learning disabled students’ reading scores are considered.

One last point: Thanks to Kentucky’s very bad former policy on testing reading, neither the NAEP nor our state test results can tell us anything about the reading ability of a very significant proportion of our learning disabled students. Very likely, thousands of the state’s learning disabled students were not given much, if any, real instruction in reading what so ever. That’s sad.

If you want to learn more about commonly encountered pitfalls in NAEP analysis, see the freedomkentucky.org wiki item: “The National Assessment of Educational Progress.”

Technical Notes: All the reading graphs were assembled using data retrieved from the Main NAEP Data Explorer.

The NAEP Science Report Card graph was cut and pasted directly from that document.