It’s a puzzle – a paradox!

The effect is so well known, it even has a name.

Kentucky Commissioner of Education Terry Holliday knows about it.

And, it explains why I consistently say that you cannot rely on overall average scores to get a good picture of the true performance of Kentucky’s public education system.

This mathematical puzzle is called “Simpson’s Paradox.” Simpson’s explains how Kentucky can actually post an above national average “all student” score even though the state’s real performance is obviously below average.

I’ve shown you before some examples of how Simpson’s Paradox confuses impressions about ACT scores, but now let’s look at another example from the recently released 2013 administration of the National Assessment of Educational Progress (NAEP).

The first table below shows actual scoring data obtained from the NAEP Data Explorer web tool on 2013 NAEP Grade 4 Reading Assessment. The table also shows the racial demographic percentages for national public schools across the nation and in Kentucky.

The scores highlighted in yellow are the “All Student” overall average scores. These show Kentucky scoring three points above the national average, just as the Kentucky Department of Education’s news release about the NAEP scores reports.

The scores for each racial group reported by the NAEP are also shown along with the percentage that racial group represented in the overall student samples for the nation and Kentucky.

For example, across the nation white students scored 231 in NAEP grade four reading while in Kentucky our whites only scored a 227. Across the nation, whites comprised just 51 percent of the public school enrollment while in Kentucky they made up a much higher 80 percent of the total.

Also note that whites notably outscored all other racial groups except for the Asian/Pacific Islanders, but that racial group is not heavily represented either in Kentucky or across the country (Note: percentages may not add to 100 due to rounding in the NAEP reporting).

Notice that when we break the scores out and compare them by race, Kentucky’s whites, blacks and students who are multi-racial were all outscored by their racial counterparts across the nation. Only Kentucky’s Asian/Pacific Islanders and Hispanics outscored the national averages for their racial counterparts in other states, and both of those groups are not heavily represented in Kentucky.

In fact, the American Indian/Alaskan Native group is so small in Kentucky that the NAEP didn’t even publish Kentucky’s scores for them.

The key point is that the vast majority of Kentucky’s students – whites, blacks and multiracial students – who collectively make up 94 percent of the total public school enrollment in the Bluegrass State, were all outscored by counterparts elsewhere.

That starts to raise strong questions about whether the Bluegrass State really did outperform the nation for fourth grade reading. Let’s make that picture even clearer.

This next table is similar to the earlier one, except I have substituted Kentucky’s racial demographics into the national public data area. I then calculated new “All Student” average scores (in the yellow shading) using the assumption that the nation had the same racial mix that we do in the Bluegrass State.

As you can see, equalizing the racial demographics dramatically shifts the picture of performance. Now, the rest of the nation overall outscores Kentucky by three points, which would be a statistically significant difference.

By the way, a similar situation is found with the eighth grade data, where instead of scoring four points above the national average, when we do a calculation like that in the second table above, Kentucky scores a statistically significant two points below the rest of the nation.

So, thanks to Simpson’s Paradox and student demographics that our education system has no control over, Kentucky only seems to score better than the nation on Grade 4 and Grade 8 NAEP Reading. The truth is that we actually perform below the national average in reading once you do an apples to apples comparison of scores. And, it is way past time for some people to own up to that unfortunate truth.