Prichard pushes the numbers too far, again, Part 2

In any state-to-state education comparison, student demographics matter

In its zeal to portray Kentucky’s public education system in the best possible light, the Prichard Committee for Academic Excellence has issued a 2012 update to its on-going “Top 20 by 2020” reports.

But, Prichard still doesn’t get this right. Prichard simply ignores advice, readily available in all federal National Assessment of Educational Progress (NAEP) report cards since 2005, that comparison of NAEP scores from state to state should consider differing student demographics and exclusion rates.

It’s not hard to understand why Prichard ignores that advice.

Consider the case of Fourth Grade NAEP Math. If we look at the officially published NAEP Scale Scores for California and Kentucky for 2011, Kentucky beats California by a statistically significant seven points.

But, California’s public schools are only 25 percent white, while Kentucky’s remain very white at 84 percent. And, whites score much higher in both states than either blacks or Hispanics do.

So, what happens to California’s overall NAEP score if we simply level the playing field by weighting California’s NAEP scores for the different races’ using the demographic makeup of Kentucky’s schools?

The answer is astonishing.

G4 NAEP Math California and Kentucky Demographics Summary Table

Instead of scoring seven points behind Kentucky, California would outscore us by seven points if that state had the same student racial demographics.

California’s educators have no control over the current immigration situation, which has flooded their schools with lots of lower-scoring minority students. And, as you can see by clicking the “Read more” link below, California has actually made amazing progress with all its significant student racial groups despite the Tsunami of non-white students that have engulfed the schools there. But, you can’t see that if you only look at the overall student scores.

One more thing – the NAEP Data Explorer shows that the eligibility rate for the federal free and reduced cost lunch program in California’s public schools for the 2011 Grade 4 Math Assessment was 58 percent while Kentucky’s was 55 percent (a difference probably not statistically significant). When we break down the NAEP performance by race for those students who are lunch eligible, Kentucky and California tie for both whites and blacks.

Kentucky’s Hispanics do outscore Hispanics in California, but there may be an unsatisfactory reason for that. Table A-5 in the 2011 NAEP Math Report Card shows an astonishingly high 98 percent of all English language learners in California participated in the NAEP. In Kentucky, a much lower proportion, just 73 percent, were tested. That will certainly inflate Kentucky’s Hispanic scores.

So, let’s not dwell on that old poverty excuse!

And, let’s get this loud and clear! In comparing state-to-state performance with the NAEP – or with any other test like the new Stanford 10 now used in Kentucky as part of the K-PREP system – it is essential to consider the scores disaggregated by race. Otherwise, Kentuckians will get an inflated sense of accomplishment in their schools.

Here are more details on this important education analysis issue.

This data, extracted using the NAEP Data Explorer, shows the results of the NAEP Grade 4 Mathematics Assessment for Kentucky and California from 1992 and 2011. It breaks the scoring down by race and includes the percentage breakdown of the racial demographics in each assessment in each state (As a note, there are plus and minus sampling errors in all these statistics, not shown).

G4 NAEP Math California and Kentucky Demographics

For example, the first line of data shows California’s overall average NAEP Grade 4 Math score was 234 in 2011. In that same year, California’s whites scored 252 but made up only 25 percent of all the public school students in the state. Blacks in California scored much lower in 2011 at 225 and made up seven percent of that state’s public school fourth grade enrollment. Scores and percentages are shown for the other racial groups that had notable populations, as well.

If we only consider the overall student scores in the table, Kentucky handily surpasses California by a statistically significant seven points on the 2011 NAEP Grade 4 Math Assessment, 241 versus 234 points. Interestingly, Kentucky’s overall student average score surpassed California by the exact same amount in 1992.

But, look what happened to white student scores. In 1992 Kentucky’s whites scored only four points below California’s whites. By 2011 California more than doubled that margin to score nine points higher than Kentucky.

Next, consider the black students’ scores. In 1992 Kentucky’s blacks scored far ahead of California’s by 18 points. Wow! Flash forward to 2011, however, and California’s blacks have totally caught up to our blacks. If we are talking about improvement over time, there is no contest. California’s blacks “whupped” us.

The picture for all of the other races listed is muddled. Kentucky simply didn’t have enough of these minority students in 1992 to allow the NAEP to publish statistically valid scores for them.

We can see that Hispanics and Asian/Pacific Islanders in Kentucky did outscore Californians from the same racial groups in 2011, but the percentages of the overall student population in both racial groups are small in Kentucky. The same situation applies to students who were reported as multi-racial, where California outscored Kentucky.

Let’s look at the changes in racial demographics in both states between 1992 and 2011 in more detail.

In Kentucky, the percentage of all public school students who are white changed very little over the nearly two-decades covered by the data. That is definitely not the case in California, where whites dropped from 50 percent to only 25 percent of the student population. The difference in California came from Hispanics, who soared from just 30 percent of the enrollment in 1992 to 54 percent in 2011. Changes for other racial groups in both states were minimal.

The key points are:

• White percentages in Kentucky changed little in Kentucky and are still very high, and

• Whites outscore blacks and Hispanics by considerable margins in Kentucky and the same is also true for whites in California versus other races in that state.

The resulting impact on overall scores gives Kentucky an unfair advantage in any simplistic comparison of NAEP scores.

To really see what is happening with NAEP (or any other state-to-state testing comparison) you must consider disaggregated data. In the case of Kentucky versus California on NAEP Grade 4 Math, once you look at those disaggregated figures, the illusion of great progress in Kentucky is quickly brought into question.