Commissioner getting our message on NAEP, sort of

Over at Kentucky Commissioner of Education Terry Holliday’s blog, he just posted these comments about nationwide performance on the National Assessment of Educational Progress (NAEP):

“All groups of children – white and non-white – have increased NAEP performance by as much as 20 to 30 points over the past 30 years. However, the overall NAEP has moved only a couple of points.”

As the commissioner’s comments point out, there is a seeming contradiction where NAEP has seen big score increases for all student racial groups while the overall average score changes have been small. The effect even has a name: “Simpson’s Paradox.”

Although the commissioner makes this sound like a recent revelation. That’s not the case.

In fact, the problem of seemingly slow overall progress in other states on the NAEP is well-known.

This problem has been well-known for years.

And, the problem has held national average and other states’ NAEP scores back much more elsewhere than in Kentucky.

Furthermore, some in Kentucky have taken advantage of the Simpson’s Paradox in NAEP to create inflated claims about Kentucky’s educational performance.

Sadly, the truth is that as soon as you disaggregate NAEP data by race, claims of great progress in Kentucky’s schools start to look a lot less credible. Consider this map based on the 2011 NAEP Grade 8 Mathematics Assessment. Clearly, in eighth grade math, our white kids have been seriously left behind. Once you consider that whites make up 84 percent of our current school enrollment, this becomes an especially disturbing situation.

G8 NAEP Math White Map

I didn’t discover the Simpson’s Paradox issue with the NAEP, but thanks to many misleading reports about Kentucky’s NAEP performance, I’ve written about this issue extensively over time (such as here, and here, for just two examples).In my writing I point out something the commissioner’s blog missed: thanks to the significant racial demographic shifts in other states, you have to be wary of people who claim Kentucky is making great progress on the NAEP compared to other states.

FACT: Kentucky has seen little shifting in the racial demographics of its schools over the past 22 years that state NAEP results have been available. In sharp contrast, national classroom demographics shifted dramatically. In some states like California, the shifts have been a Tsunami – with the percentage of whites tested dropping from around 50 percent in the early 1990s to only around 25 percent currently. Those shifts elsewhere make it much harder for the nation and other states to show progress in overall state NAEP scores.

This is why the Bluegrass Institute has correctly pointed out for years that state NAEP data has to be disaggregated and compared by race to develop any true idea about real educational performance. Simplistic comparisons only of overall scores for all students (such as discussed here) simply fall prey to Simpson’s Paradox.

If you want to learn more, check out this freedomkentucky.org article and its section about pitfalls in interpreting NAEP scores. This article was initially generated in 2009. It’s too bad our commissioner of education, the National Assessment Governing Board and too many who write about NAEP in Kentucky apparently never read it.